Human- Versus Machine Learning-Based Triage Using Digitalized Patient Histories in Primary Care: Comparative Study.

Area of Science:

• Digital Health
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Smartphone technology enables patients to report symptoms digitally prior to primary care visits.
• Machine learning (ML) can analyze these digital symptom data to aid in triage decisions for appropriate care levels.

Purpose of the Study:

• To evaluate the interrater reliability between human primary care physicians (PCPs) and an automated ML-based triage system.
• To compare ML triage performance against a consensus of expert physician judgment.

Main Methods:

• A naive Bayes ML model was developed to classify digital medical histories for urgent physical examination needs.
• The ML model was tested on 300 reports, with classifications compared against the majority vote of 5 PCPs.
• Interrater reliability was assessed using Cohen κ and percentage agreement.

Main Results:

• The ML model showed low interrater reliability (Cohen κ = 0.17) compared to the physician panel's majority vote.
• Physician agreement among themselves was also low (Cohen κ = 0.2), and intrarater reliability was moderate (Cohen κ = 0.55).
• Agreement was higher for non-urgent cases (74%) than for urgent cases (42%).

Conclusions:

• Low interrater and intrarater agreement among PCPs in triage decisions hinders the use of human judgment as a reliable benchmark for ML automation.
• The variability in physician decision-making presents a challenge for developing and validating automated triage systems in primary care.