Validation of the Enhanced Opioid Identification and Co-occurring Disorders Algorithms

Area of Science:

• Health Informatics
• Public Health Research
• Data Science in Healthcare

Background:

• Accurate identification of hospital encounters for specific health conditions is crucial for public health surveillance and resource allocation.
• Existing algorithms often combine medical codes and natural language processing (NLP) for enhanced accuracy.
• The 2016 National Hospital Care Survey (NHCS) provides a valuable dataset for evaluating such algorithms.

Purpose of the Study:

• To validate the reliability of two algorithms designed to identify hospital encounters related to opioid involvement, opioid overdose, substance use disorders, and mental health issues.
• To assess algorithm performance using established metrics like F-score and Matthews correlation coefficient.
• To identify areas for improvement in algorithm design and data collection for future NHCS analyses.

Main Methods:

• A validation study was conducted using a stratified sample of 900 hospital encounters from the 2016 NHCS.
• Medical record abstraction was performed by trained abstractors to determine the presence of opioid involvement, overdose, substance use disorder, and mental health issues.
• Algorithm outputs were compared against abstractor determinations to calculate performance metrics (F-score, Matthews correlation coefficient).

Main Results:

• Algorithm performance varied significantly across the targeted conditions and metrics.
• The opioid-involvement algorithm demonstrated the highest performance with an F-score of 0.95.
• The opioid overdose algorithm showed the lowest performance (F-score of 0.48), with Matthews correlation coefficient indicating generally poorer performance across all algorithms.
• Identified causes for discrepancies included overly broad code/keyword inclusions and incomplete survey data.

Conclusions:

• The validation study identified strengths and weaknesses of the developed algorithms, guiding future refinement.
• Algorithm performance highlights the critical importance of data completeness in large-scale healthcare surveys.
• Findings provide a foundation for improving future analyses of the NHCS and similar datasets.