Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current...

Area of Science:

• Digital health
• Health informatics
• Medical AI

Background:

• Web-based symptom checkers (SCs) are rapidly growing without standardized quality assurance.
• Vignette studies are common for SC evaluation, but accuracy is a composite metric influenced by tester variability.
• Existing methods lack widely accepted criteria for assessing SC performance.

Purpose of the Study:

• To evaluate the impact of tester variability on SC outcome accuracy using clinical vignettes.
• To explore the feasibility of measuring isolated aspects of SC performance.
• To improve the reliability and generalizability of SC performance assessments.

Main Methods:

• Assessed Healthily's SC with 114 vignettes across three tester groups (free, partially free, restricted instructions).
• Calculated κ statistics for agreement on outcome condition and triage.
• Measured crude and adjusted accuracy against a gold standard, with adjusted accuracy refined by a review process.
• Assessed symptom comprehension feasibility using variations of 51 chief complaints across three SCs.

Main Results:

• Intertester agreement improved with increased standardization (free: 0.49-0.51; partially free: 0.66-0.66; restricted: 0.71-0.72).
• Restricted group accuracy averaged 50.6% (SD 5.35%), with adjusted accuracy at 56.1%.
• Symptom comprehension assessment was feasible, with scores ranging from 52.9% to 68%.

Conclusions:

• Standardizing vignette testing significantly enhances intertester agreement but does not eliminate tester-dependent variability.
• Composite accuracy measures in vignette studies are limited by tester variability and small sample sizes.
• Measuring isolated SC performance aspects, like symptom comprehension, provides more reliable assessments.
• An adjusted accuracy measure and isolated metrics are recommended for future SC performance evaluations.