The effect of sample size and response grading on estimates of the scorer's general beta reliability coefficient for diverse types of transformed and raw scores

Abstract

Objectives: The study aimed to examine the effect of sample size and response grading on estimates of the scorer’s general beta reliability coefficient for different types of transformed and raw scores using real data measuring student attitudes toward mathematics in TIMSS in 2019. Methodology: The study sample consisted of 5,117 male and female fourth-grade students in the Sultanate of Oman. Three types of data were generated by modifying the number of response grading categories in the scale (four-point grading, three-point grading, and binary grading). For each grading type, 100 subsamples were randomly selected using three different subsample sizes (30, 200, and 1,000). Five types of transformed scores were calculated: raw score, percentile rank, standard score, first-TIMES score, and second-TIMES score, and the coefficient was calculated. The overall beta reliability of the scorers on the five transformed scores for all data was compared using descriptive statistics, bias ratios, and root mean square error (RMSE) values.

Results: The study reached a high degree of accuracy in the estimates of the general beta reliability coefficient for the scorers across all transformed grades, as the average values of the reliability coefficient estimates for all grades in the random samples were very close to the reliability coefficients of the main sample. The bias ratios and RMSE values were small and close to zero, including for the small sample size (30).

Conclusion: The study recommended expanding the use of the general beta reliability coefficient to calculate the stability of all types of transformed and raw scores in research studies and various assessment tools.