A Comparative Study of Item Response Theory Models for Mixed Discrete-Continuous Responses

Area of Science:

• Educational Measurement
• Artificial Intelligence in Education
• Psychometrics

Background:

• Language proficiency assessments are crucial for education and careers.
• AI integration enables complex item types like dictation tasks with mixed response distributions.
• Existing measurement models may not adequately capture these unique response features.

Purpose of the Study:

• To evaluate novel measurement models for AI-driven language assessments with mixed discrete-continuous response features.
• To assess the performance of zero-and-one-inflated extensions of Beta, Simplex, and Samejima's Continuous item response models.
• To incorporate collateral information using latent regression for improved parameter estimation.

Main Methods:

• Evaluation of extended Beta, Simplex, and Samejima's Continuous item response models.
• Application of latent regression to incorporate collateral information.
• Comparison of model performance using item and person parameters and out-of-sample predictive accuracy.

Main Results:

• All evaluated models yielded highly correlated item and person parameters.
• The Beta item response model demonstrated superior out-of-sample predictive accuracy.
• A significant challenge identified is the lack of established benchmarks for model and item fit for these novel models.

Conclusions:

• Novel measurement models, particularly the Beta item response model, are effective for AI-driven language assessments with mixed response distributions.
• Further research is essential to develop benchmarks for evaluating the fit of these innovative models.
• Establishing reliability and validity benchmarks is critical for real-world application of these advanced assessment tools.