Setting Discontinue Rules for Progressive Tests: A Practical and Transparent Toolkit

Area of Science:

• Psychometrics
• Educational Measurement
• Test Theory

Background:

• Progressive tests administer items by increasing difficulty, using discontinue rules to manage participant burden and resource use.
• Current discontinue rules lack a systematic, evidence-based framework for optimization.

Purpose of the Study:

• To propose a transparent approach for evaluating the impact of different discontinue rules on test accuracy and efficiency.
• To facilitate informed decision-making for setting appropriate discontinue rules in progressive testing.

Main Methods:

• Developed 'A-E charts' to visualize the trade-offs between accuracy and efficiency for various discontinue rules.
• Applied discontinue rules retroactively to normative item response data to generate A-E charts.

Main Results:

• Demonstrated that a universal discontinue rule is unlikely to be optimal across all testing scenarios.
• Showcased that the optimal discontinue rule is contingent upon the desired efficiency-accuracy balance and specific test context.

Conclusions:

• The proposed A-E charting approach offers a pragmatic method for practitioners and test developers to evaluate and select discontinue rules.
• Encourages a systematic re-evaluation of existing discontinue rules to enhance test efficiency and maintain accuracy.