Reporting Valid and Reliable Overall Scores and Domain Scores Using Bi-Factor Model

Area of Science:

• Psychometrics
• Educational Measurement
• Statistical Modeling

Background:

• Increasing demand for accurate diagnostic information in large-scale testing programs.
• Limited research on reliable reporting of total and domain scores using bi-factor models.

Purpose of the Study:

• To introduce and compare six novel methods for reporting overall and domain scores derived from bi-factor models.
• To evaluate the performance of these methods against Yao's multidimensional item response theory (MIRT) method.

Main Methods:

• Development of six weighted composite score reporting methods for bi-factor models.
• Comparison using simulated data (varying test length, dimensionality, correlation, sample size) and empirical data.
• Evaluation metrics included score accuracy, reliability, and parameter recovery.

Main Results:

• Bifactor-M4 and Bifactor-M6 demonstrated superior accuracy and reliability for overall and domain scores across most conditions.
• These methods excelled particularly with longer tests, higher dimensional correlations, and more dimensions.
• Bifactor-M4 showed the best recovery of true ability parameters; Bifactor-M2 performed poorly on overall scores; Bifactor-M1 yielded the worst estimations.

Conclusions:

• Bifactor-M4 and Bifactor-M6 are recommended for reporting scores from bi-factor models due to their accuracy and reliability.
• The choice of weighting method significantly impacts score estimation quality.
• Further research should explore the practical implementation and interpretation of these advanced scoring methods.