Large Language Models for Psychiatric Phenotype Extraction from Electronic Health Records

Area of Science:

• Psychiatry
• Computational Linguistics
• Bioinformatics

Background:

• Accurate clinical phenotyping from electronic health records (EHRs) is crucial for psychiatric research.
• Extracting symptom-level information from free-text clinical notes is challenging but essential.

Purpose of the Study:

• To evaluate the performance of 11 open-source generative large language models (LLMs) for detecting 109 psychiatric phenotypes from Spanish clinical text.
• To compare LLM performance against traditional natural language processing (tNLP) methods.
• To develop and validate a fine-tuned LLM for psychiatric phenotyping.

Main Methods:

• Tested 11 generative LLMs on annotated EHR notes from a Colombian psychiatric clinic.
• Evaluated LLMs in zero-shot, few-shot, and fine-tuned settings.
• Compared LLM performance to a tNLP method and validated fine-tuned models on external data.

Main Results:

• Base LLM performance was poor to moderate; fine-tuning significantly improved performance (macro-F1 from 0.2-0.74 to 0.75-0.86).
• Several fine-tuned LLMs outperformed the tNLP method, with 100 phenotypes reliably detected (F1>0.8).
• A synthetic dataset enabled creation of 'Mistral-small-psych' (macro-F1=0.79), which generalized well to external data.

Conclusions:

• Domain-specific fine-tuning enhances LLM capabilities for psychiatric phenotyping.
• Mistral-small-psych offers a valuable tool for accurate, large-scale psychiatric phenotyping in Spanish.
• This work supports the advancement of global precision psychiatry through improved EHR data utilization.