Related Experiment Video
Updated: Jul 2, 2026
07:14
Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
Published on: December 23, 2025
Iterative Multidisciplinary Development and Evaluation of a Patient-Facing SDoH Chatbot Using Synthetic Data
Anna M Maw1, Alexander Lupi1,2, Rachel Johnson-Koenke1
1Department of Emergency Medicine, University of Colorado Anschutz Medical Campus, Aurora, US.
JMIR Formative Research
|June 30, 2026
Summary
This study developed and evaluated a social determinants of health (SDoH) chatbot using synthetic data and case simulations. The approach refined chatbot performance and evaluation metrics before clinical deployment.
Area of Science:
- Artificial Intelligence in Healthcare
- Digital Health Interventions
- Health Informatics
Background:
- Inconsistent social determinants of health (SDoH) data collection impacts patient outcomes.
- Large language model (LLM)-powered chatbots show potential for scalable SDoH data collection.
- Lack of rigorous evaluation methods for patient-facing SDoH chatbots hinders clinical adoption.
Purpose of the Study:
- To describe an efficient, iterative, multidisciplinary method for developing and evaluating a patient-facing SDoH chatbot.
- To optimize chatbot performance and evaluation rubric using synthetic data and case simulation prior to clinical deployment.
Main Methods:
- Adapted a 10-criterion evaluation rubric from healthcare AI frameworks.
- Applied rubric to 27 synthetic SDoH clinical scenarios role-played by a clinical social worker.
- Multidisciplinary expert team (social worker, nurse practitioner, physician) rated chatbot-patient interactions.
- Utilized quantitative (percent agreement, Fleiss' κ) and qualitative analysis for iterative refinement.
Main Results:
- Chatbot achieved high ratings for accurate interpretation (98%), communication quality (99%), and adaptive questioning (99%).
- Lower performance observed in domain focus (51%), data capture completeness (59%), and safety (69%).
- Qualitative feedback refined rubric definitions, particularly for 'safety,' focusing on emergency recognition.
Conclusions:
- A formative feasibility approach using synthetic case simulation effectively refines patient-facing SDoH chatbots and evaluation rubrics.
- Future work will involve external raters, patient input, repeated testing, and prospective clinical evaluation.