Examination of ChatGPT's Performance as a Data Analysis Tool

Area of Science:

• Psychometrics
• Artificial Intelligence
• Computational Statistics

Background:

• OpenAI's ChatGPT is a widely adopted AI conversational tool.
• The application of advanced AI models in complex statistical analyses like EFA is an emerging area of research.
• Assessing the reliability and validity of AI tools for scientific data analysis is crucial.

Purpose of the Study:

• To evaluate the performance of ChatGPT-4o as a tool for exploratory factor analysis (EFA).
• To compare ChatGPT-4o's data analysis results with traditional statistical software (R) under various simulated data conditions.
• To identify the strengths and limitations of using AI conversational tools in psychometric data analysis.

Main Methods:

• Simulated datasets were generated varying factors such as data distribution, response categories, sample size, test length, factor loading, and measurement models.
• ChatGPT-4o was used to perform EFA on the simulated data twice, with a one-week interval, using identical prompts.
• Key EFA metrics including Kaiser-Meyer-Olkin (KMO) values, total variance explained, factor loadings, and the number of factors (using Kaiser-Guttman, Hull, and empirical Kaiser criteria) were calculated and compared against R results.

Main Results:

• Analyses conducted using ChatGPT-4o yielded results consistent with those obtained using R code.
• ChatGPT-4o demonstrated strong performance in computational aspects of EFA, such as calculating KMO values, total variance explained, and factor loadings.
• While the number of estimated factors was consistent, biases were noted in ChatGPT-4o's analysis of multidimensional structures.

Conclusions:

• ChatGPT-4o shows promise as a computational tool for specific EFA procedures, particularly those not requiring subjective researcher judgment.
• Researchers should exercise caution when utilizing ChatGPT-4o for EFA involving complex, multidimensional data structures due to potential biases.
• Further research is needed to refine AI models for nuanced statistical decision-making in psychometrics.