Evaluating digital triage symptom checker with historical triage-related adverse events

Online symptom checkers are increasingly used for diagnostic support and triage. However, evidence on their performance and evaluations with real-world data remains limited.

The aim of this study was to evaluate the performance of a digital symptom checker using clinical vignettes derived from real-world cases that had previously been incorrectly triaged.

A patient-facing, rule-based digital symptom checker used in Swedish primary care was assessed in this study. Vignettes were constructed from cases reported to the Swedish Health and Social Care Inspectorate due to erroneous telephone triage. We hypothesized that the digital symptom checker could provide appropriate triage for these cases. Seven physicians independently simulated patients by entering symptoms in the symptom checker based on each vignette. Triage outcomes were assessed against the Swedish National Triage Guidelines (RGS), evaluating the accuracy and the safety of the triage recommendation.

A total of 69 unique vignettes yielded 483 individual trials. After excluding 93 trials due to significant deviations from the original vignette description (adding or omitting symptoms), 390 trials were included in the primary analysis. The symptom checker achieved 91% accuracy (95% CI 88-94%) and 94% safety (95% CI 91-96%).

The symptom checker demonstrated high accuracy and safety when triaging a subset of vignettes based on real-world cases that had been previously erroneously triaged. This study also highlights the difficulties of using vignettes when evaluating symptom checkers. To our knowledge, this is the first study to evaluate such systems using vignettes based on actual patient cases with known triage errors.