Polytomous Testlet Response Models for Technology-Enhanced Innovative Items: Implications on Model Fit and Trait...

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Innovative items in technology-enhanced assessments often feature polytomous scoring within testlets.
• Existing measurement models may not adequately capture the complexities of these polytomous testlet items.
• Practical models are needed to accurately describe and score such items.

Purpose of the Study:

• To evaluate and compare the performance of four measurement models for polytomous items administered in testlets.
• To identify the most suitable model based on criteria such as model fit, parameter recovery, and classification accuracy.
• To provide guidelines for selecting appropriate measurement models for polytomous testlet items.

Main Methods:

• Comparison of four models: generalized partial credit model (GPCM), testlet-as-a-polytomous-item model (TPIM), random-effect testlet model (RTM), and fixed-effect testlet model (FTM).
• Empirical evaluation using data simulating GPCM, FTM, and RTM conditions.
• Assessment of models based on relative and absolute fit, testlet effect significance, parameter recovery, and classification accuracy.

Main Results:

• Model performance varied significantly based on testlet effect type, size, and trait estimator.
• GPCM and FTM yielded optimal results when testlets had no or fixed effects.
• RTM showed the best model fit for random interaction effects but performance varied with trait estimation methods.
• The effectiveness of RTM was most apparent with strong random effects and Bayesian estimation.

Conclusions:

• The choice of measurement model for polytomous testlet items depends critically on the nature and magnitude of testlet effects.
• Simpler models like GPCM and FTM are often sufficient and perform comparably or better in many scenarios.
• Polytomous scoring of testlet items has limited practical utility.
• Guidelines are provided for selecting the most appropriate measurement model for polytomous innovative items in testlet-based assessments.