Modeling Rapid Guessing Behaviors in Computer-Based Testlet Items

Area of Science:

• Psychometrics
• Educational Measurement
• Cognitive Psychology

Background:

• Traditional item response theory (IRT) models assume item independence, which is violated in testlets with common stimuli.
• Test-takers may engage in rapid guessing (RG) due to lack of motivation, knowledge, or time, especially in computer-based tests.
• Ignoring item dependence within testlets and RG behavior can lead to biased measurement results.

Purpose of the Study:

• To propose a novel mixture testlet IRT model that incorporates response times to simultaneously model item responses and RG behavior.
• To evaluate the performance of the proposed model in accurately recovering item and person parameters.
• To assess the impact of ignoring RG behavior on parameter estimation in computer-based testlet items.

Main Methods:

• Development of a mixture testlet IRT model integrating item responses and response time data.
• Two simulation studies using Markov chain Monte Carlo (MCMC) estimation via JAGS to test model recovery and bias.
• Application of the proposed model and a traditional IRT model to real data from a computer-based language test.

Main Results:

• The proposed mixture testlet IRT model demonstrated good recovery of item and person parameters in simulation studies.
• Ignoring RG behavior led to significant biases, including overestimated item difficulties and underestimated time intensities.
• Parameter estimates from the model applied to real test data showed differences consistent with simulation findings.

Conclusions:

• The proposed mixture testlet IRT model effectively addresses rapid guessing in computer-based testlets by utilizing response time data.
• Failing to account for rapid guessing can introduce substantial bias into item and person parameter estimates.
• This new model offers a more accurate approach to analyzing data from computer-based tests with complex item structures and potential speededness.