Two-Stage Extraction of Clinical Course Information From Psychiatric Discharge Summaries Using Fine-Tuned Large...

Area of Science:

• Natural Language Processing in Healthcare
• Psychiatric Informatics
• Machine Learning for Clinical Data

Background:

• Clinical course data (onset, recurrence, hospitalization) is vital for psychiatric care but difficult to extract from unstructured narratives.
• Manual and rule-based extraction methods are labor-intensive and lack scalability.
• Large language models (LLMs) offer a potential solution for flexible information extraction from sensitive psychiatric records.

Purpose of the Study:

• To evaluate the performance of fine-tuned LLMs in automatically extracting temporal and clinical course information from psychiatric discharge summaries.
• To assess the utility of LLMs for transforming unstructured clinical narratives into structured data.

Main Methods:

• Analysis of 500 psychiatric discharge summaries with manual annotation of clinical events and temporal information.
• Fine-tuning of four open-source LLMs (LLaMA, MentaLLaMA, OpenBioLLM, Mistral) using low-rank adaptation.
• Development and evaluation of a 2-stage framework for sentence-level extraction followed by chart-level prediction of clinical course features.
• Performance assessment using precision, recall, F1-score, accuracy, and mean absolute error (MAE).

Main Results:

• Mistral achieved the highest F1-scores at the sentence level for clinical event extraction (e.g., symptom/episode detection: 0.925) and temporal information.
• The 2-stage framework demonstrated significant benefit for first-episode onset prediction, outperforming direct and joint extraction approaches.
• Mistral achieved the highest onset accuracy (0.772) using the 2-stage framework.
• Performance varied for other chart-level features, with different models excelling in episode count, hospitalization count, and most recent hospitalization prediction.

Conclusions:

• Fine-tuned, open-source LLMs can effectively extract temporal and longitudinal disease course information from psychiatric discharge summaries.
• The 2-stage framework is particularly beneficial for predicting first-episode onset and shows competitive performance for other clinical course features.
• LLMs hold promise for converting heterogeneous psychiatric narratives into structured data, aiding research and clinical decision support.