Enhancing substance use detection in clinical notes with large language models

Area of Science:

• Medical Informatics
• Natural Language Processing
• Substance Use Research

Background:

• Electronic health records (EHRs) contain valuable patient data, but substance use behaviors are often hidden in unstructured clinical notes.
• Varied terminology, negation, and contextual nuances complicate accurate identification of substance use from EHRs.

Purpose of the Study:

• To develop and evaluate large language models (LLMs) for detecting eight substance use categories within EHR discharge summaries.
• To create a large, annotated dataset from MIMIC-III/IV discharge summaries for drug detection research.
• To support systemic substance use surveillance through improved automated detection.

Main Methods:

• Utilized MIMIC-III/IV discharge summaries to construct a comprehensive drug detection dataset.
• Investigated the performance of various LLMs in zero-shot, few-shot, and fine-tuning scenarios.
• Evaluated models on their ability to detect specific substance use categories, including prescription opioid misuse and polysubstance use.

Main Results:

• A fine-tuned LLM, Llama-DrugDetector-70B, demonstrated superior performance in substance use detection.
• Achieved high F1-scores (≥0.95) for most individual substance categories.
• Showed strong performance on complex tasks: prescription opioid misuse (F1=0.815) and polysubstance use (F1=0.917).

Conclusions:

• LLMs significantly enhance the accuracy of identifying substance use behaviors from unstructured EHR data.
• The developed Llama-DrugDetector-70B model shows promise for clinical decision support and large-scale substance use surveillance.
• Further research is needed to address the scalability of LLM-based detection methods in real-world clinical settings.