The accuracy vs. coverage trade-off in patient-facing diagnosis models

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• A significant portion of adults use the internet for medical diagnosis.
• Online symptom checkers are becoming prevalent tools for self-diagnosis.
• These tools utilize diagnosis models akin to clinical decision support systems.

Purpose of the Study:

• To investigate the performance-coverage trade-off in diagnostic models for online symptom checkers.
• To be the first study to analyze this specific trade-off in diagnostic AI.
• To inform the development of more accurate and comprehensive diagnostic tools.

Main Methods:

• Learning diagnostic models with varying disease coverage using electronic health record (EHR) data.
• Evaluating model performance based on accuracy metrics.
• Comparing the performance of different model complexities, including linear models and neural networks.

Main Results:

• A 1% decrease in top-3 diagnostic accuracy was observed for every 10 additional diseases included in the model's coverage.
• Model complexity did not significantly impact performance.
• Linear models achieved comparable performance to more complex neural networks.

Conclusions:

• There is an inherent trade-off between the diagnostic coverage and accuracy of online symptom checkers.
• Simpler models like linear regression can be as effective as complex neural networks for this task.
• Findings suggest a need to balance breadth of conditions with diagnostic precision in developing these tools.