A validity-guided workflow for robust large language model research in psychology.

Area of Science:

• Psychology
• Artificial Intelligence
• Computational Social Science

Background:

• Large language models (LLMs) are increasingly used in psychological and behavioral research.
• Significant measurement unreliability has been observed, leading to
• measurement phantoms
• which are statistical artifacts mistaken for psychological phenomena.

Purpose of the Study:

• To address the threat of measurement unreliability in LLM-based psychological research.
• To present a six-stage workflow integrating psychometrics and causal inference to ensure research validity.
• To guide researchers in distinguishing genuine computational phenomena from artifacts.

Main Methods:

• Proposed a dual-validity framework combining psychometrics with causal inference.
• Outlined a six-stage workflow for validating LLM research: defining goals, validating instruments, controlling confounds, transparent execution, appropriate analysis, and reporting.
• Illustrated the workflow with an LLM selfhood evaluation example.

Main Results:

• Identified "measurement phantoms" as a critical threat to the validity of LLM research.
• Demonstrated how the proposed workflow can systematically validate computational instruments.
• Showcased the ability to differentiate genuine computational phenomena from artifacts through systematic validation.

Conclusions:

• A robust workflow is essential for establishing validated computational instruments and transparent practices in AI psychology.
• Implementing this workflow can build a more reliable empirical foundation for research using LLMs.
• Systematic validation is key to advancing the field of AI psychology and ensuring research integrity.