Challenging the norm: Length of exams determined by classification accuracy or reliability

Area of Science:

• Medical Education
• Psychometrics
• Assessment Science

Background:

• Traditional medical education exams often rely on reliability indices to determine test length.
• However, the primary goal of these exams is to ensure defensible pass-fail decisions, a purpose not fully met by reliability alone.

Purpose of the Study:

• To challenge the use of reliability indices for test length decisions in medical education.
• To propose classification accuracy as a more appropriate metric for pass-fail decisions.
• To empirically demonstrate that classification accuracy leads to shorter recommended test lengths compared to reliability.

Main Methods:

• Analysis of re-sampled test data from undergraduate medical knowledge exams (N=52,500 datasets).
• Systematic variation of cut-scores and test lengths in generated synthetic exams.
• Estimation of both reliability and classification accuracy indices for each dataset.

Main Results:

• Classification accuracy, unlike reliability, varies with the cut-score for pass-fail decisions.
• Reliability and classification accuracy show different relationships with test length.
• Optimal test length for reliability is ~100 items, irrespective of pass rates.
• For classification accuracy, 50 items can achieve 95% accuracy with ≤5% failure rates.

Conclusions:

• Advocate for the adoption of classification accuracy in medical education assessments, complementing existing reliability measures.
• Implementing classification accuracy can reduce the assessment burden on candidates and developers.
• Emphasize the importance of considering false positive and false negative decisions in pass/fail classifications.