Assessing the Properties and Functioning of Model-Based Sum Scores in Multidimensional Measures With Local Item...

Area of Science:

• Psychometrics
• Statistical Modeling

Background:

• Correlated residuals are a common issue in assessing noncognitive attributes using factor analysis (FA).
• Previous research focused on the structural impact of correlated residuals, with less attention to their effect on score accuracy in extended FA solutions.
• Existing measures assess reliability and information in sum scores from unidimensional FA, but extensions to more complex models are needed.

Purpose of the Study:

• To extend measures of reliability, factor saturation, and information to second-order factor analytic solutions with a single general factor.
• To adapt the added-value principle to second-order FA scenarios with local dependencies.
• To develop a new coefficient for assessing added value, including effect size and confidence intervals.

Main Methods:

• Development of new coefficients to assess score reliability and relative efficiency at subscale and total scale levels.
• Extension of the added-value principle to second-order FA models.
• Implementation of proposed measures in a freely available R program.

Main Results:

• The proposed methods allow for the assessment of score reliability and relative efficiency in second-order FA.
• New coefficients provide insights into the added value of subscale scores versus total scores in prediction.
• An empirical example demonstrates the distortions caused by correlated residuals and interprets the proposed measures.

Conclusions:

• The developed measures and coefficients enhance the assessment of score quality in the presence of correlated residuals in second-order FA.
• The R program facilitates the practical application of these advanced psychometric techniques.
• Understanding the impact of correlated residuals is crucial for accurate interpretation of noncognitive attribute assessments.