Assessing data gathering of chatbot based symptom checkers - a clinical vignettes study

Area of Science:

• Healthcare Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• Healthcare systems face increasing burdens from population growth, aging demographics, and pandemic-related pressures.
• This overload negatively impacts healthcare quality and patient outcomes.
• Intelligent self-assessment tools, or symptom-checkers, are emerging as a potential solution to integrate into healthcare systems.

Purpose of the Study:

• To evaluate the data-gathering capabilities of currently available chatbot symptom-checkers.
• To assess how effectively these tools simulate the information-gathering process of a medical interview.

Main Methods:

• Eight symptom-checker platforms were evaluated using 28 clinical vignettes.
• Medical students simulated patient roles, interacting with the symptom-checkers.
• Key metrics included recall rates (findings retrieved/total findings) and efficiency rates (findings retrieved/questions asked).

Main Results:

• The overall recall rate across all symptom-checkers was 32% for pertinent findings.
• Kahun platform achieved the highest recall rate at 51%.
• The overall efficiency rate was 46%, with Kahun also demonstrating the highest efficiency (74%).

Conclusions:

• The data-gathering performance of current symptom-checkers is suboptimal.
• Kahun emerged as the top-performing tool in terms of both recall and efficiency.
• Further development is needed to enhance the clinical utility of these AI-driven tools.