Related Experiment Video
Updated: Aug 9, 2026
Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
Published on: December 23, 2025
Iterative Multidisciplinary Development and Evaluation of a Patient-Facing Social Determinants of Health Chatbot
Anna M Maw1,2, Alexander Lupi3,4, Rachel Johnson-Koenke5
1Division of Hospital Medicine, University of Colorado School of Medicine, 12401 East 17th Avenue, Mailstop F-782, Aurora, CO, 80045, United States, 1 720 848 4289.
This study developed and evaluated a social determinants of health (SDoH) chatbot using synthetic data, showing high performance in communication but areas for improvement in data capture and safety. The iterative approach refined both the chatbot and its evaluation rubric for future clinical use.
Area of Science:
- Health Informatics
- Artificial Intelligence in Healthcare
- Social Determinants of Health Research
Background:
- Inconsistent systematic collection of social determinants of health (SDoH) data impacts patient outcomes.
- Large language model (LLM) chatbots show potential for scalable SDoH data collection.
- There is a lack of rigorous evaluation methods for patient-facing SDoH chatbot applications.
Purpose of the Study:
- To describe an efficient, iterative, multidisciplinary approach for developing and evaluating a patient-facing SDoH chatbot.
- To optimize chatbot performance and evaluation rubric using synthetic data and case simulation before clinical deployment.
Main Methods:
- Adapted a 10-criterion evaluation rubric from healthcare AI frameworks.
- Applied rubric to 27 synthetic clinical scenarios role-played by a clinical social worker.
- Multidisciplinary expert team (social worker, nurse practitioner, physician) rated chatbot-patient interactions.
- Used quantitative analysis (percent agreement, Fleiss κ) and qualitative synthesis of rater feedback for iterative refinement.
Main Results:
- Chatbot received high ratings for accurate interpretation (agreement=0.98%), communication quality, cultural sensitivity (agreement=0.99%), and adaptive questioning (agreement=0.99%).
- Lower performance observed in domain focus/completeness (agreement=0.51%), data capture completeness (agreement=0.59%), and safety (agreement=0.69%).
- Qualitative feedback informed rubric refinement, clarifying safety to focus on emergency recognition.
Conclusions:
- A formative feasibility approach using synthetic case simulation enables iterative refinement of patient-facing SDoH chatbots and their evaluation rubrics.
- Future work will involve external raters, patient stakeholders, and prospective clinical evaluation.
