Trifactor Models for Multiple-Ratings Data.

Area of Science:

• Educational Measurement
• Psychometrics
• Statistical Modeling

Background:

• Multiple-ratings data, common in educational assessments, presents unique statistical challenges.
• Existing models often assume hierarchical structures, which may not capture complex rater-item interactions.
• Accurate assessment of student performance requires robust models that handle data complexities.

Purpose of the Study:

• To extend the trifactor model to accommodate cross-classified data structures, moving beyond hierarchical assumptions.
• To investigate the impact of data missingness and inter-rater variability on assessment accuracy.
• To validate the enhanced trifactor model using real-world international large-scale assessment data.

Main Methods:

• Extension of the trifactor model to a cross-classified framework for item and rater data.
• Conducting simulation studies to assess model performance under varying degrees of data missingness and rater differences.
• Application and evaluation of the enhanced model on empirical data from an international large-scale assessment.

Main Results:

• The extended trifactor model effectively handles cross-classified data structures in multiple-ratings assessments.
• Simulation results highlight the significant impact of data missingness and unmodeled rater differences on score reliability.
• Empirical analysis demonstrates the practical utility and robustness of the enhanced trifactor model.

Conclusions:

• The enhanced trifactor model provides a more accurate and flexible approach for analyzing multiple-ratings data in large-scale assessments.
• Accounting for cross-classifications and rater variability is crucial for valid score interpretation.
• This work offers improved statistical tools for educational measurement researchers and practitioners.