Comparing the Performance of Eight Item Preknowledge Detection Statistics

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Analysis

Background:

• Item preknowledge, where examinees access test items beforehand, invalidates scores and compromises testing programs.
• Detecting item preknowledge is challenging due to unknown aberrant examinees, locations, and item subsets.
• Existing statistical methods for detecting compromised items yield a 'suspicious subset' with inherent uncertainty.

Purpose of the Study:

• To evaluate the impact of uncertainty in the suspicious subset on the performance of eight statistical detection methods.
• To understand how various factors influence the effectiveness of these detection statistics.

Main Methods:

• Performance assessment using receiver operating characteristic (ROC) curves.
• Computer simulations to model the effects of uncertainty and independent variables on statistic performance.

Main Results:

• Uncertainty in the suspicious subset significantly affects the performance of statistical detection methods.
• The impact of uncertainty varies depending on factors like test type and aberrant examinee distribution.
• Simulations quantified the performance degradation across different statistics under varying uncertainty levels.

Conclusions:

• The performance of statistical methods for detecting compromised items is sensitive to uncertainty in the identified suspicious item subset.
• Understanding these performance impacts is vital for developing robust detection strategies in high-stakes testing.
• Further research should focus on mitigating the effects of uncertainty in item preknowledge detection.