Misclassification Produced by Rapid-Guessing Identification Methods and Their Suitability Under Various Conditions

Area of Science:

• Educational Measurement
• Psychometrics
• Data Science

Background:

• Response Time Threshold Methods (RTTMs) are crucial for detecting rapid-guessing behavior (RG) in low-stakes assessments.
• Challenges include misclassifications due to overlapping response times and disagreement on optimal methods.

Purpose of the Study:

• To evaluate the performance of five RTTMs under varying conditions.
• To identify the most robust method for detecting RG in simulation studies.

Main Methods:

• Simulated item responses and response times from one-component (no RG) and two-component mixture models (with RG).
• Varied distribution, item, and person parameters.
• Evaluated five RTTMs: mixture lognormal (MLN), cumulative proportion (CUMP), and mixture response time quantile (MRTQ), among others.

Main Results:

• MLN demonstrated superior robustness in identifying RG, estimating precise thresholds closest to optimal misclassification points.
• CUMP was less robust but accurate when successful; it was the only method to set thresholds when RG was absent.
• Most methods were conservative, except MRTQ.

Conclusions:

• MLN is recommended as a robust method for RG identification, especially when bimodality is subtle.
• A practical six-step RG identification process is proposed, integrating mixture modeling (MLN or MRTQ) with CUMP.