Natural Language Response Formats for Assessing Depression and Worry With Large Language Models: A Sequential...

Area of Science:

• Natural Language Processing
• Computational Psychiatry
• Machine Learning in Healthcare

Background:

• Large language models (LLMs) can quantify mental health from user descriptions.
• Previous LLM analyses often combined diverse text response types.
• The differential impact of response formats on LLM analysis validity and reliability is not well understood.

Purpose of the Study:

• To develop and compare the validity and reliability of LLM-based mental health scoring across different response formats.
• To investigate the performance of LLMs using closed-ended (word lists) to open-ended (text) response formats.
• To assess the external validity of LLM-derived scores against clinical outcomes like sick leave.

Main Methods:

• Developed four response formats: word lists, descriptive words, phrases, and free text.
• Trained machine learning models on word embeddings from participant responses (N=963) to predict depression/worry scores.
• Employed a Sequential Evaluation with Model Pre-Registration (SEMPR) design, testing pre-registered models on a prospective sample (N=145).
• Evaluated concurrent, incremental, face, discriminant, and external validity, alongside prospective and test-retest reliability.

Main Results:

• Pre-registered models demonstrated strong validity and reliability, achieving high accuracy (r=0.60-0.79) in the prospective sample.
• LLM analyses showed external validity, correlating with self-reported sick leave and healthcare visits.
• The free-text response format yielded the strongest correlations with external outcomes, matching or exceeding traditional rating scales in most cases (9/12).

Conclusions:

• LLM-based mental health assessment is valid and reliable across various response formats.
• The choice of response format can be tailored to specific clinical needs and desired outcomes.
• LLMs offer a flexible and accurate tool for mental health quantification, with text formats showing particular promise for external validity.