Toward a better judgment of item relevance in progress testing

Area of Science:

• Educational Measurement
• Psychometrics
• Assessment Validity

Background:

• Item relevance is crucial for ensuring the quality and validity of educational assessments.
• The concept of "item relevance" lacked a standardized operational definition, hindering consistent evaluation.
• A novel rubric was developed to operationalize and systematically assess item relevance.

Purpose of the Study:

• To evaluate the impact of a newly developed rubric on the assessment of item relevance.
• To determine the influence of the rubric on inter-rater agreement in item evaluation.
• To assess the rubric's effect on the inclusion decisions for progress test items.

Main Methods:

• A 5-criteria rubric was used by an item review committee (RC) and students, teachers, and alumni (STA) to reassess 50 progress test (PT) items.
• Paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression were employed for item-level analysis.
• Generalizability analysis was conducted at the rater level to assess reliability within and between groups.

Main Results:

• The proportion of items deemed relevant for inclusion by the RC significantly decreased from 1.00 to 0.72.
• High inter-rater agreement (ICC > 0.7) was observed between the RC and STA.
• A strong correlation (0.89) was found between item relevance and inclusion decisions, consistent across groups.

Conclusions:

• The rubric enforces a more rigorous evaluation of item appropriateness for progress tests.
• Implementation of the rubric enhances consensus and agreement among diverse stakeholders.
• The rubric is a valuable tool for improving the acceptability and validity of progress tests.