From Prompts to Constructs: A Dual-Validity Framework for Large Language Model Research in Psychology

Area of Science:

• Psychology
• Artificial Intelligence
• Computational Linguistics

Background:

• Large language models (LLMs) are increasingly used in psychological research as tools and subjects.
• Current studies often apply human measures to LLMs without validating their reliability or interpretability.
• This risks mistaking statistical regularities for genuine psychological phenomena (measurement phantoms).

Purpose of the Study:

• To propose a methodological framework for robust AI-driven psychological research.
• To address the challenges of applying human-centric measures to LLMs.
• To guide the development of reliable and interpretable LLM research in psychology.

Main Methods:

• Review of existing methodologies in psychometrics and causal inference.
• Development of a dual-validity framework for AI psychological research.
• Analysis of evidentiary requirements based on scientific ambition (tool use, behavioral characterization, human simulation, cognitive modeling).

Main Results:

• A dual-validity framework is proposed, integrating psychometric validation and causal inference.
• Evidentiary demands increase with the scientific goal, from simple classification to cognitive modeling.
• Human measures do not automatically apply to LLMs; computational analogs of psychological constructs are needed.

Conclusions:

• Robust AI psychological research requires integrating psychometric validation and causal inference.
• Claims about LLM psychological simulation or cognitive mechanisms necessitate rigorous construct validity and experimental controls.
• Future progress hinges on developing specific computational approaches rather than relying on direct application of human measures.