Scalable, context-sensitive psychiatric assessment with large language models and brief diaries

Area of Science:

• Computational psychiatry
• Natural Language Processing (NLP) in mental health
• Psychometric validation of AI tools

Background:

• Accurate psychiatric assessment necessitates understanding individual experiences within their psychosocial context.
• Traditional clinical interviews are effective but time and resource-intensive, leading to reliance on narrow, decontextualized surveys.
• This creates a scalability-comprehensiveness tradeoff, hindering psychopathology research and treatment.

Purpose of the Study:

• To explore the use of large language models (LLMs) for scoring psychopathology from brief personal narratives.
• To offer a low-burden, context-sensitive solution to the limitations of current psychiatric assessment methods.
• To evaluate the validity of LLM-derived psychopathology scores.

Main Methods:

• 108 participants completed daily ~1-minute audio diaries for 2 weeks.
• Six LLMs were employed to score a wide range of psychopathology domains from transcribed audio diaries.
• Convergent, discriminant, concurrent, and clinical validity of LLM ratings were assessed against self-report and clinical interview measures.

Main Results:

• LLM ratings demonstrated strong convergent and discriminant validity, correlating most highly with corresponding self-report domains at both between-person and within-person levels.
• LLM and self-report ratings showed similar associations with external variables, with minor exceptions for specific domains.
• All LLM-rated psychopathology domains showed significant relationships with those ascertained via clinical interview.

Conclusions:

• LLMs can effectively assess major forms of psychopathology using minimal audio data (minutes).
• LLM-based scoring of open-ended narratives provides a scalable and portable method for psychiatric assessment.
• This approach translates idiographic (individual) diagnostic data into standardized psychiatric evaluations.