Improving drug identification in overdose death surveillance by using clinical natural language processing models

Area of Science:

• Public Health
• Computational Linguistics
• Forensic Science

Background:

• Drug-related deaths, particularly fentanyl-induced, are rising in the US, necessitating efficient surveillance.
• Current methods rely on manual coding of coroner reports into ICD-10, causing data delays and loss.
• Existing natural language processing (NLP) applications for overdose surveillance have shown limitations.

Purpose of the Study:

• To evaluate and compare various NLP models for classifying drug involvement in overdose deaths from unstructured text.
• To assess the performance of traditional machine learning, BERT-based models, and large language models (LLMs).
• To determine the most accurate and scalable NLP approach for enhancing overdose surveillance.

Main Methods:

• Utilized a dataset of 35,433 US death records from 2020 for training and internal testing.
• Performed external validation on a separate dataset of 3335 records from 2023-2024.
• Compared traditional classifiers, fine-tuned Bidirectional Encoder Representations from Transformers (BERT) and BioClinicalBERT, and decoder-only LLMs (Qwen 3, Llama 3).
• Assessed performance using macro-averaged F1 scores and 95% confidence intervals.

Main Results:

• Fine-tuned BioClinicalBERT models achieved near-perfect performance (macro F1 ≥0.998) on internal data.
• External validation demonstrated BioClinicalBERT's robustness (macro F1 = 0.966), outperforming other models.
• NLP models significantly outperformed conventional machine learning and general-domain BERT.

Conclusions:

• Fine-tuned clinical NLP models, like BioClinicalBERT, provide a highly accurate and scalable solution for classifying overdose deaths from free-text reports.
• These NLP methods can substantially accelerate surveillance, overcoming manual coding limitations.
• The study supports near real-time detection of emerging substance use trends through advanced NLP techniques.