Hybrid Threshold-Based Sequential Procedures for Detecting Compromised Items in a Computerized Adaptive Testing...

Area of Science:

• Psychometrics
• Educational Measurement
• Computerized Adaptive Testing (CAT)

Background:

• Ensuring test item security is crucial in standardized testing.
• Computerized adaptive testing (CAT) requires robust methods for detecting compromised items.
• Traditional methods may lack sufficient power or efficiency in real-time CAT environments.

Purpose of the Study:

• To apply sequential procedures for detecting compromised items in a real operational item pool within a variable-length CAT.
• To propose and evaluate a hybrid threshold approach to enhance the detection power of sequential procedures while controlling Type I error rates.
• To investigate the impact of simulation factors on the power rate and lag time of the detection procedure.

Main Methods:

• Utilized classical test theory and item response theory.
• Applied sequential procedures to a variable-length CAT item pool.
• Developed and tested a hybrid threshold approach combining local and global thresholds.
• Conducted simulation studies with various factors and a case study on a real item pool.

Main Results:

• The probability increment (p-increment) was the most significant factor influencing the detection of compromised items.
• The hybrid threshold approach, particularly the local threshold, improved power rates and reduced lag times when the p-increment was small.
• The proposed procedures demonstrated applicability to real-world CAT item pools.

Conclusions:

• The hybrid threshold approach offers an effective strategy for real-time detection of compromised items in CAT.
• Practitioners can leverage these findings to implement enhanced security monitoring in live CAT administrations.
• Further research can explore optimizing threshold parameters for different testing contexts.