Diagnostic Accuracy of Artificial Intelligence in Virtual Primary Care

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Primary care faces challenges in timely and accurate disease diagnosis.
• Artificial intelligence (AI) offers potential solutions for augmenting clinical decision-making.

Purpose of the Study:

• To evaluate the diagnostic accuracy of AI-generated clinical diagnoses in a primary care setting.
• To assess the agreement between AI diagnoses and human healthcare providers.

Main Methods:

• Retrospective chart review of 102,059 virtual primary care encounters.
• Analysis of AI-generated differential diagnoses against provider final diagnoses and adjudicator consensus.
• Evaluation of AI diagnostic agreement across various diagnoses, symptoms, demographics, and provider experience levels.

Main Results:

• Providers agreed with an AI diagnosis in 84.2% of cases and the top-ranked AI diagnosis in 60.9%.
• High agreement (≥90%) was observed for 69% of diagnoses and 50% of presenting symptoms.
• AI diagnostic accuracy showed minimal variation across patient demographics and provider experience, improving with retraining.

Conclusions:

• AI demonstrates high diagnostic agreement with providers in virtual primary care settings.
• AI has the potential to enhance primary care disease diagnosis and patient triage.
• Continuous model retraining can further improve AI diagnostic performance over time.