Item Response Theory Modeling for Examinee-selected Items with Rater Effect

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Examinee-selected item (ESI) designs in large-scale testing present challenges with missing data.
• Data in ESI designs can be missing not at random (MNAR) due to examinees selecting easier items.
• Existing item response theory (IRT) models often fail to account for rater effects in ESI designs.

Purpose of the Study:

• To develop novel IRT models addressing both MNAR data and rater severity in ESI designs.
• To adapt existing estimation methods for the complexities of ESI designs with rater effects.
• To evaluate the performance of new models against conventional IRT approaches.

Main Methods:

• Development of a new IRT model integrating MNAR data and rater severity.
• Adaptation of conditional maximum likelihood estimation and pairwise estimation for ESI designs.
• Simulation studies comparing new methods with conventional IRT models.

Main Results:

• The new IRT model demonstrated good parameter recovery.
• Conditional maximum likelihood and pairwise estimation methods were effective when Rasch models fit the data.
• Conventional IRT models produced biased parameter estimates when ignoring MNAR data or rater severity.

Conclusions:

• The developed IRT models offer a robust solution for ESI designs with MNAR data and rater effects.
• New estimation methods provide reliable parameter estimates in specific conditions.
• The findings highlight the limitations of conventional IRT models in complex testing designs.