Identifying quality responses using an analysis of response times: the RTcutoff function in R

Area of Science:

• Psychometrics
• Educational Measurement
• Data Science

Background:

• Response time data in assessments can indicate participant engagement levels.
• Distinguishing between thoughtful engagement and rapid guessing is crucial for accurate interpretation of test scores.
• Existing methods may not fully integrate response time and accuracy to identify engagement patterns.

Purpose of the Study:

• To develop and validate an R function for analyzing response time data.
• To establish thresholds for differentiating between engaged and disengaged test-taking behaviors.
• To visualize response time profiles relative to sample performance.

Main Methods:

• Utilized the cumulative proportion correct (CUMP) approach to estimate item-specific time thresholds.
• Developed an R function incorporating CUMP and response time profiles (mean + 1SD).
• Applied the function to reading comprehension data from Saudi Arabian students (PIRLS 2021).

Main Results:

• The developed RTcutoff function effectively identified variable engagement patterns.
• Results highlighted the necessity of the function for assessing person-score and test validity.
• Demonstrated the function's utility in differentiating thoughtful engagement from rapid responding or guessing.

Conclusions:

• The RTcutoff function is essential for improving measurement precision in educational assessments.
• Integrating response time and accuracy enhances the validity of person-score interpretations.
• The CUMP approach provides a robust method for establishing engagement thresholds.