Using Item Scores and Response Times to Detect Item Compromise in Computerized Adaptive Testing

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing

Background:

• Sequential procedures are effective for real-time detection of compromised items in computerized adaptive testing (CAT).
• Item response theory (IRT) models are widely used in CAT for accurate ability estimation.
• Integrating response times (RTs) with item scores can enhance the detection of aberrant response behavior.

Purpose of the Study:

• To propose and evaluate three novel IRT-based sequential procedures for detecting compromised items in CAT.
• To investigate the utility of combining item scores and RTs for improved detection accuracy.
• To compare the performance of different sequential procedures under various simulation conditions.

Main Methods:

• Developed three sequential procedures based on IRT: (1) score or RT statistic extreme, (2) score and RT statistic extreme, (3) combined score and RT statistic extreme.
• Simulated data under various conditions to assess the performance of each procedure.
• Evaluated procedures based on false-positive rate, true-positive rate, and lag time.

Main Results:

• The third procedure, utilizing a combined score and RT-based statistic, demonstrated the most promising results.
• This combined approach achieved a favorable balance between the false-positive and true-positive rates.
• The proposed procedures, particularly the third one, produced relatively short lag times across diverse simulation scenarios.

Conclusions:

• The combined score and RT-based sequential procedure is highly effective for real-time detection of compromised items in CAT.
• This method offers a practical and efficient solution for maintaining test integrity in adaptive testing environments.
• Future research should explore the application of these procedures in real-world testing contexts.