Developing an Approach Based on Behavioral Objectives to Estimate the Cut-off Score: A Comparison with Angoff's Method

Abstract

This study aimed at developing an approach based on
behavioral objectives to estimate the cut-off score compared with
Angoff's method, which determines the cut-off score based on testitems.
In order to achieve this aim a content domain was selected, well
defined, and analyzed by translation this content into a set of
behavioral objectives. Every objective in the set was followed by a
group of multiple-choice items, which measure this objective to enable
the raters to estimate the cut-off score for every objective. Two test
forms were constructed using two procedures of selecting items for
each form based on item-objective congruence for one form, and
random selection for the other one.A total group of 30 raters
estimated the cut-score of each objective and item in the two test
forms. The raters were classified according to their educational
qualifications and work status into three equal groups. Two of the
three groups estimated the cut-off scores using Angoff's method, two
times or rounds over a period of two weeks. A sample of 171 ninth
grade students were tested immediately after they had completed
studying the unit in order to calculate the statistical indices of the
students and the items, for validating the cut-off scores and for
comparison purposes.The results of the study indicated that the cut-off
score based on objectives was (0.56) in the two raters' rounds, while it
was (0.67) for the first form using Angoff's method in the first round,
(0.66) in the second round, and (0.63) for the second form in the two
rounds. Furthermore, all correlation coefficients between the raters'
estimations in the two rounds were high (0.93; 0.96) and were
statistically significant at (α = 0.01). The results also indicated that
there was no statistically significant difference between the reliability
coefficients of the two methods (objective-based and test-item-based)
of estimating cut-off scores.The results of the study also indicated that
there was no statistically significant difference between the estimated
cut-off scores by the raters in the three groups based on the
behavioral objectives. The two methods were high and equivalent
statistically of two reliability indices (Hit rate, and Kappa coefficient);
but they were significantly different in their ability to predict a
student’s school achievement in favor of the objective-based method.
The study concludes that the objective-based method is superior in
estimating cut-off scores compared with the most frequently used testitem-
based method; it is more efficient in predictive validity, more
powerful and universal due to the strong-grounded early steps
starting specifying the content domain, and the population of
behavioral objectives representing this domain