Validation of the Enhanced Opioid Identification and Co-occurring Disorders Algorithms

Objectives This report documents the results of a validation study conducted to assess the reliability of two algorithms applied to the 2016 National Hospital Care Survey. One algorithm identifies opioid-involved and opioid overdose hospital encounters, and the other identifies encounters with patients that have substance use disorders and selected mental health issues. These algorithms use both medical codes and natural language processing to identify encounters. Methods To validate the algorithms, medical record abstraction was performed on a stratified sample of 900 hospital encounters from the 2016 National Hospital Care Survey. The abstractors recorded their determinations of opioid involvement, opioid overdose, substance use disorder, and mental health issues on a standard form. Abstractors' determinations were compared with algorithm output to assess the overall performance using F-score and Matthews correlation coefficient. The latter provided a secondary measure of performance. The 2016 National Hospital Care Survey data are unweighted and not nationally representative. Results Overall algorithm performance varied by topic and by metric. The opioid-involvement algorithm achieved the highest performance, performing well with an F-score of 0.95, followed by the substance use disorder algorithm (F-score of 0.79), the mental health issues algorithm (F-score of 0.68), and the opioid overdose algorithm (F-score of 0.48). Assessment by Matthews correlation coefficient indicated an overall poorer level of performance, ranging from a high of 0.57 for the mental health issues algorithm to a low of 0.33 for the opioid-involvement algorithm. The causes of false positives and false negatives likewise varied, including both overly broad code and keyword inclusions as well as incompleteness of data submitted to the National Hospital Care Survey. Conclusion The validation study illustrates which aspects of the developed algorithms performed well and which aspects should be altered or discarded in future iterations. It further emphasizes the importance of data completeness, therefore laying the groundwork for improvements to future survey analyses.