Characterizing Sampling Variability for Item Response Theory Scale Scores in a Fixed-Parameter Calibrated Projection...

Area of Science:

• Psychometrics
• Educational Measurement
• Statistics

Background:

• Current scale linking practices often ignore sampling variability in item parameters.
• This oversight leads to underestimated uncertainty in projected Item Response Theory (IRT) scale scores.
• Accurate uncertainty estimation is crucial for reliable score interpretation and decision-making.

Purpose of the Study:

• To develop and evaluate a novel method for adjusting posterior standard deviations of IRT scale scores.
• To address the underestimation of uncertainty caused by neglecting carry-over sampling variability in scale linking.
• To improve the accuracy of score uncertainty estimation in psychometric applications.

Main Methods:

• Application of a Multiple Imputation (MI) approach to incorporate sampling variability of item parameters.
• Utilizing Restricted Recalibration (RR) with fixed item parameters for conditional estimation of scale relationships.
• Conducting a simulation study to assess the impact of carry-over sampling variability under diverse conditions.

Main Results:

• The combination of Restricted Recalibration (RR) and Multiple Imputation (MI) effectively accounts for carry-over sampling variability.
• The proposed method provides more accurate estimates of uncertainty in Item Response Theory (IRT) scale scores compared to traditional methods.
• Demonstrated practical application of the RR-MI method using real-world psychometric data.

Conclusions:

• The proposed Multiple Imputation (MI) method, combined with Restricted Recalibration (RR), offers a statistically sound approach to scale linking.
• This method enhances the reliability and accuracy of uncertainty estimation for Item Response Theory (IRT) scale scores.
• The findings have significant implications for improving the precision of score comparisons across different assessments.