Explaining GPTs' schema of depression: A machine behavior analysis

Area of Science:

• Artificial Intelligence in Mental Health
• Clinical Psychology
• Computational Psychiatry

Background:

• Large language models (LLMs) are increasingly used in mental health, yet their internal understanding of disorders like depression is unclear.
• Assessing the internal schema of LLMs is crucial for safe and effective clinical deployment.

Purpose of the Study:

• To decode the internal schema of depressive symptoms within GPT-4 and GPT-5 using measurement theory.
• To evaluate the reliability and validity of LLM-based depression assessment.
• To compare the internal symptom networks of GPT-4 and GPT-5.

Main Methods:

• Applied contemporary measurement theory to analyze symptom interrelationships in GPT-4 and GPT-5.
• Assessed convergent validity against standard instruments and expert judgments.
• Mapped internal symptom networks and calculated correlation coefficients.

Main Results:

• GPT-4 demonstrated strong convergent validity (r = .70-.81) with existing depression measures.
• LLM symptom networks largely aligned with clinical literature (r = .23-.78), but underemphasized suicidality and overemphasized psychomotor symptoms.
• Identified novel hypotheses regarding symptom mechanisms, such as sleep/fatigue being broadly influenced.

Conclusions:

• LLMs can reliably assess depression constructs, offering insights into clinical applications.
• Understanding LLM internal schemas is vital for interpreting their mental health assessments.
• Findings provide an empirical basis for evaluating LLMs in mental healthcare across different models and disorders.