Automating detection of diagnostic error of infectious diseases using machine learning

Area of Science:

• Medical Informatics
• Machine Learning in Healthcare
• Patient Safety

Background:

• Diagnostic errors contribute significantly to patient morbidity and mortality.
• Current methods for identifying diagnostic errors are manual, costly, and not real-time.
• Electronic health record (EHR) data offers potential for automated misdiagnosis detection.

Purpose of the Study:

• To develop and validate an automated approach for identifying diagnostic divergence in emergency department (ED) infectious disease cases.
• To measure diagnostic deviation by comparing predicted diagnoses with documented diagnoses, weighted by mortality risk.
• To assess the utility of machine learning models in detecting potential misdiagnoses at scale.

Main Methods:

• Trained two machine learning models to predict infectious disease and mortality using the first 24 hours of EHR data.
• Analyzed 6.5 million ED visits across over 100 emergency departments.
• Validated the automated approach against manual chart reviews by clinicians using Spearman rank correlation.

Main Results:

• The infectious disease model achieved a Macro F1 score of 86.7 and AUROC of 90.6-94.7.
• The mortality model achieved a Macro F1 score of 97.6 and AUROC of 89.1.
• Positive correlations (0.231-0.358) were observed between the automated metric and manual clinician reviews.

Conclusions:

• The proposed automated approach shows promise for identifying diagnostic errors in real-world clinical settings.
• This method can serve as a valuable tool for clinicians to detect potential misdiagnoses.
• Future work should incorporate data structure and natural language processing for enhanced accuracy and explainability.