Perils and opportunities in using large language models in psychological research

Area of Science:

• Psychological Research
• Artificial Intelligence
• Computational Social Science

Background:

• Large language models (LLMs) are increasingly explored for psychological research applications.
• Concerns exist regarding the uncritical adoption of LLMs, termed "GPTology", without fully understanding their limitations and risks.
• Existing general guidelines may not fully address the specific nuances of LLM use in psychological contexts.

Purpose of the Study:

• To investigate the current limitations, ethical implications, and potential of LLMs within psychological research.
• To analyze the concrete impact of LLMs in various empirical psychological studies.
• To advocate for responsible and methodologically sound integration of LLMs in the field.

Main Methods:

• Review of current LLM capabilities and limitations relevant to psychological research.
• Analysis of empirical studies employing LLMs in psychology.
• Ethical considerations and risk assessment of LLM application.
• Exploration of transparent and open methods for AI-generated data inference.

Main Results:

• LLMs present significant limitations and risks when applied to psychological research, particularly concerning global psychological diversity.
• Treating LLMs, especially in zero-shot settings, as universal text-analysis solutions is cautioned against.
• The opaque nature of LLMs necessitates the development of transparent, open methods for reliable and reproducible AI-driven inference.

Conclusions:

• Acknowledging LLM utility for tasks like text annotation and understanding human psychology is important.
• Diversifying human samples and expanding psychology's methodological toolbox are crucial.
• Promoting an inclusive, generalizable science requires countering homogenization and over-reliance on LLMs.