Interpreting psychiatric digital phenotyping data with large language models: a preliminary analysis.

Area of Science:

• Artificial Intelligence in Healthcare
• Digital Phenotyping
• Computational Psychiatry

Background:

• Digital phenotyping offers passive monitoring for behavioral health but struggles with translating complex data into clinical insights.
• Digital navigators interpret this data, but workforce limitations hinder scalability.

Purpose of the Study:

• To systematically evaluate large language model (LLM) performance in interpreting simulated psychiatric digital phenotyping data.
• To establish baseline accuracy metrics for LLMs in this emerging clinical application.

Main Methods:

• GPT-4o and GPT-3.5-turbo were evaluated on over 153 test cases with simulated digital phenotyping data.
• Test cases mimicked scenarios, timeframes, and data quality levels used for training human digital navigators.
• Model performance was assessed by comparing their ability to identify clinical patterns against human experts.

Main Results:

• GPT-4o achieved 52% accuracy, significantly outperforming GPT-3.5-turbo (12%) in interpreting psychiatric digital phenotyping data.
• GPT-4o showed high accuracy for worsening depression (100%) and anxiety (83%), but lower for other patterns (e.g., 6% for improved symptoms with increased home time).
• Accuracy decreased with lower data quality (39%) and shorter timeframes (43% for 3 weeks vs. 60% for 3 months).

Conclusions:

• GPT-4o's 52% accuracy provides a baseline for LLM interpretation of digital phenotyping data, though human oversight is essential due to performance gaps and hallucinations.
• Significant variations in LLM performance across models, data quality, and clinical scenarios necessitate careful implementation strategies.
• LLMs can potentially augment human digital navigators, addressing workforce shortages while maintaining clinical oversight in digital phenotyping.