Overfactoring in rating scale data: A comparison between factor analysis and item response theory.

Area of Science:

• Educational Measurement
• Psychological Measurement
• Psychometrics
• Factor Analysis

Background:

• Traditional factor analysis models often assume normally distributed data.
• Skewed response distributions in educational and psychological measurement attenuate correlations.
• Attenuated correlations can lead to an overestimation of factors in linear factor models.

Purpose of the Study:

• To investigate the problem of overfactoring in factor analysis with skewed data.
• To compare the performance of five different factor analysis approaches under skewed conditions.
• To identify reliable methods for estimating the number of factors when data are not normally distributed.

Main Methods:

• A simulation study was conducted to evaluate factor analysis techniques.
• Five methods were compared: maximum-likelihood factor analysis (FA), categorical factor analysis (FAC) with ML and WLS estimation, Satorra-Bentler corrected chi-square, and Samejima's graded response model (GRM).
• Goodness-of-fit criteria included likelihood-ratio chi-square, parallel analysis (PA), and categorical parallel analysis (CPA).

Main Results:

• Maximum-likelihood estimation resulted in overfactoring for both linear and categorical factor models when variables were skewed.
• The Satorra-Bentler method and the graded response model (GRM) demonstrated superior reliability in estimating the correct number of factors.
• Categorical factor analysis with weighted least squares (WLS) also showed some improvement over standard ML estimation.

Conclusions:

• Standard maximum-likelihood estimation is susceptible to overfactoring with skewed psychological and educational measurement data.
• The Satorra-Bentler correction and the graded response model (GRM) are recommended as robust alternatives for factor number estimation.
• Researchers should consider these advanced methods when dealing with non-normally distributed item responses to ensure accurate factor structures.