Development and Validation of Machine Models Using Natural Language Processing to Classify Substances Involved in...

Area of Science:

• Computational epidemiology
• Public health informatics
• Natural language processing in medicine

Background:

• Overdose deaths are a leading cause of mortality in the US.
• Existing surveillance data reporting suffers from considerable delays.
• Timely data is crucial for effective public health interventions.

Purpose of the Study:

• To develop and evaluate natural language processing (NLP) and machine learning (ML) algorithms for automated classification of substance-related deaths.
• To assess the performance of different NLP/ML approaches in identifying specific substances linked to overdose fatalities.
• To reduce the time lag in reporting critical overdose surveillance data.

Main Methods:

• A diagnostic study analyzed 35,433 unstructured death records from 10 US health jurisdictions (2020).
• Compared three feature representation methods: TF-IDF, GloVe, and CUI embeddings.
• Trained and selected ML classifiers based on F-scores, with final evaluation on a hold-out test set.

Main Results:

• The most common substances identified were any opioid (16%), fentanyl (13%), alcohol (8%), cocaine (6%), and methamphetamine (5%).
• Concept Unique Identifier (CUI) embeddings performed comparably or better than TF-IDF and GloVe for most substances.
• ML classifiers achieved excellent performance for opioids, fentanyl, heroin, cocaine, methamphetamine, and alcohol; benzodiazepine classification was suboptimal.

Conclusions:

• NLP/ML algorithms demonstrate high diagnostic accuracy for classifying overdose-related substances.
• These automated methods can significantly decrease the reporting lag for overdose surveillance.
• Integration into public health workflows is recommended for improved real-time monitoring of the overdose crisis.