Improving Drug Identification in Overdose Death Surveillance using Large Language Models.

Area of Science:

• Public Health Surveillance
• Computational Linguistics
• Data Science

Background:

• Drug-related deaths, primarily fentanyl-driven, necessitate rapid and precise surveillance in the U.S.
• Current methods relying on manual coding of coroner reports into ICD-10 classifications cause data delays and loss.
• Existing natural language processing (NLP) applications for overdose surveillance have shown limitations.

Purpose of the Study:

• To evaluate and compare various NLP models for classifying specific drug involvement from unstructured death certificate text.
• To assess the performance of traditional machine learning, general-domain BERT, and large language models (LLMs) against fine-tuned clinical NLP models.
• To determine the efficacy of NLP in automating and enhancing overdose surveillance data extraction.

Main Methods:

• Utilized a dataset of 35,433 U.S. death records from 2020 for training and internal testing.
• Performed external validation on a separate dataset of 3,335 records from 2023-2024.
• Compared traditional classifiers, Bidirectional Encoder Representations from Transformers (BERT), BioClinicalBERT, Qwen 3, and Llama 3 using macro-averaged F1 scores.

Main Results:

• Fine-tuned BioClinicalBERT models achieved near-perfect performance (macro F1 >=0.998) on the internal test set.
• External validation demonstrated the robustness of BioClinicalBERT (macro F1=0.966), outperforming other evaluated models.
• NLP models significantly outperformed conventional machine learning and general-domain BERT and LLMs.

Conclusions:

• Fine-tuned clinical NLP models, like BioClinicalBERT, provide a highly accurate and scalable solution for classifying drug-involved overdose deaths from free-text reports.
• These NLP methods can substantially accelerate surveillance workflows, surpassing the limitations of manual ICD-10 coding.
• The approach supports near real-time detection of emerging substance use trends, improving public health response.