Disengaged response behavior when the response button is blocked: Evaluation of a micro-intervention

Area of Science:

• Educational Measurement
• Psychometrics
• Human-Computer Interaction

Background:

• Disengaged participants in large-scale assessments may rapidly guess or skip items, compromising score validity.
• Accidental navigation and rapid responses are challenges in computer-based assessments.
• Effective methods are needed to identify and mitigate invalid response behaviors.

Purpose of the Study:

• To evaluate a micro-intervention (2-second response block) designed to prevent accidental navigation and rapid responses.
• To analyze the impact of this intervention on participant response behavior.
• To assess the intervention's utility in improving the detection of rapid guessing and omissions.

Main Methods:

• Analysis of log event sequences from a linear computer-based assessment using a finite-state machine approach.
• Classification of responses into different classes based on event sequences.
• Application of post hoc methods using response time thresholds to validate classifications and detect rapid responding.

Main Results:

• The 2-second response block interfered with rapid responses but did not significantly alter overall participant behavior.
• Rapid guesses and omissions were distinguishable from accidental clicks using log event data.
• The blocked response improved post hoc detection of rapid responding, particularly for responses near time thresholds.

Conclusions:

• While the micro-intervention had limited impact on behavior, it enhanced the accuracy of identifying rapid response strategies.
• Blocking responses may help prevent accidental item skipping, potentially increasing its adoption.
• When analyzing assessment data for rapid responses, incorporating additional log data is crucial for accurate interpretation.