Calibrating Multidimensional Assessments With Structural Missingness: An Application of a Multiple-Group Higher-Order...

Area of Science:

• Educational Measurement
• Psychometrics
• Item Response Theory

Background:

• Educational constructs are increasingly complex, requiring measurement at both general and subdomain levels.
• Current methods often necessitate large item banks or report scores separately, limiting practical assessment.
• Simultaneous reporting of general and subdomain scores is desirable but challenging.

Purpose of the Study:

• To propose and evaluate a multiple group hierarchical ordered item response theory (HO-IRT) model with structural missingness for simultaneous score reporting.
• To investigate a novel application scenario using a NEAT (North, East, South, West) design with both representative and non-representative anchor tests.
• To explore the parameter recovery of HO-IRT models when using a non-representative anchor test.

Main Methods:

• Utilized a multiple group HO-IRT model with structural missingness.
• Employed a NEAT design with both representative and non-representative anchor tests.
• Conducted Monte Carlo simulations to assess parameter recovery and Root Mean Square Error (RMSE).
• Addressed missing data using a full-information maximum likelihood approach.

Main Results:

• The study demonstrated that a non-representative anchor test can yield comparable RMSE to a representative anchor test.
• Parameter recovery was found to be robust even with a moderate correlation between higher- and lower-order factors.
• The proposed HO-IRT model effectively controls assessment length while reporting both general and subdomain scores.

Conclusions:

• Multiple group HO-IRT models offer a viable solution for simultaneously reporting general and subdomain scores for complex educational constructs.
• The use of non-representative anchor tests in a NEAT design is a practical alternative when construct definitions evolve.
• This approach enhances efficiency in educational measurement without compromising score accuracy.