Enhancing Substance Use Detection in Clinical Notes with Large Language Models

Area of Science:

• Medical Informatics
• Natural Language Processing
• Public Health Surveillance

Background:

• Identifying substance use in electronic health records (EHRs) is difficult due to unstructured clinical notes, varied terminology, and negation.
• Accurate detection is crucial for patient care, clinical decision support, and public health surveillance of substance use behaviors.

Purpose of the Study:

• To develop and evaluate large language models (LLMs) for detecting eight substance use categories within EHR discharge summaries.
• To create a large, annotated dataset for drug detection to support systemic substance use surveillance.

Main Methods:

• Utilized MIMIC-III/IV discharge summaries to construct an annotated drug detection dataset.
• Investigated the performance of multiple LLMs in zero-shot, few-shot, and fine-tuning settings.
• Evaluated models on detecting individual substance use, prescription opioid misuse, and polysubstance use.

Main Results:

• A fine-tuned LLM, Llama-DrugDetector-70B, demonstrated superior performance.
• Achieved near-perfect F1-scores (>=0.95) for most individual substance use categories.
• Showed strong performance for prescription opioid misuse (F1=0.815) and polysubstance use (F1=0.917).

Conclusions:

• LLMs significantly enhance the detection of substance use behaviors in EHRs.
• Fine-tuned LLMs show promise for clinical decision support and research in substance use surveillance.
• Further research is needed to address the scalability of LLM applications in this domain.