Assessing the Alignment of Large Language Models With Human Values for Mental Health Integration: Cross-Sectional...

Area of Science:

• Artificial Intelligence Ethics
• Computational Social Science
• Psychological Measurement

Background:

• Large language models (LLMs) offer potential for mental health, but opaque alignment processes may embed biases.
• Evaluating embedded values in LLMs is crucial for ethical AI development and deployment.
• Schwartz's theory of basic values (STBV) provides a framework for assessing value orientations.

Purpose of the Study:

• To evaluate if the STBV can measure value-like constructs in leading LLMs.
• To determine if LLMs exhibit distinct value-like patterns compared to humans and each other.

Main Methods:

• Four LLMs (Bard, Claude 2, GPT-3.5, GPT-4) completed the Portrait Values Questionnaire-Revised (PVQ-RR).
• LLM responses were analyzed for reliability and validity, then compared to human data from 53,472 individuals across 49 nations.
• Statistical tests compared value profiles between LLMs and against human population data.

Main Results:

• The PVQ-RR reliably quantified value-like constructs in LLMs, but significant divergence from human value profiles emerged.
• LLMs prioritized universalism and self-direction, de-emphasizing achievement, power, and security relative to humans.
• Distinct value profiles differentiated the four LLMs and predicted their responses to mental health dilemmas.

Conclusions:

• The STBV effectively mapped value-like infrastructure in LLMs, revealing substantial divergence from human values.
• Embedded biases pose ethical risks for LLM integration into mental health applications, requiring safeguards.
• Standardizing alignment processes is essential to capture cultural diversity and ensure equitable AI deployment in mental healthcare.