Factors Shaping Trust and Satisfaction With AI Medical Chatbots: A Mixed Methods Vignette Survey of Caregivers...

As artificial intelligence (AI) chatbots become an increasingly common source of quick medical guidance, it is important to understand whether their responses meet users' needs and support well-informed health decisions. Yet, existing evaluation frameworks rely primarily on expert-defined evaluation dimensions that have not been empirically validated with end users. It remains unclear whether these frameworks capture the criteria people actually use when judging a response to be useful, trustworthy, and satisfying, or which evaluation dimensions matter most to users in practice.

This study empirically examined how commonly used dimensions such as Accuracy and Comprehensiveness shape caregivers' satisfaction with AI-generated answers to pediatric health questions. We further investigated what expectations and communication needs may be overlooked by current evaluation frameworks.

We conducted a mixed methods vignette survey with 191 caregivers recruited through Prolific. Participants evaluated GPT-4o responses to a set of clinician-approved pediatric health questions across 8 dimensions and rated overall satisfaction. We quantified the influence of each dimension on overall satisfaction using a Cumulative Link Mixed Model and performed an inductive thematic analysis of open-ended comments to identify gaps in established frameworks.

A total of 191 caregivers evaluated 1146 chatbot responses. Initially, caregivers rated Accuracy and Credibility as the most important dimensions. However, Cumulative Link Mixed Model analysis identified Usefulness as the strongest driver of overall satisfaction (odds ratio [OR] 2.53, 95% CI 1.95-3.27; P<.001), followed by Thoroughness (OR 2.15, 95% CI 1.69-2.73; P<.001). Comprehensiveness did not significantly influence satisfaction (OR 1.05, 95% CI 0.82-1.33; P=.71). Qualitative feedback helped explain this: participants frequently criticized responses as "too long" and preferred concise, actionable guidance. Empathy/Warmth was significantly associated with overall satisfaction (OR 1.48, 95% CI 1.27-1.72; P<.001) but elicited polarized reactions (Van der Eijk agreement μ=0.38): some caregivers valued emotional support, while others found AI-generated empathy insincere and undermined credibility. Medical disclaimers increased trust in higher-risk situations but reduced confidence in lower-risk scenarios.

Existing evaluation frameworks only partially capture how caregivers assess medical chatbots. Caregivers valued actionable guidance, credible references, and clear reasoning over lengthy, exhaustive detail. Rather than passively receiving dense information, they preferred an interactive style in which the chatbot proactively proposed follow-up suggestions, helping them steer the conversation toward their specific needs. Reactions to empathetic language and medical disclaimers were context-dependent: features that built trust in some situations could seem insincere, excessive, or unnecessary in others. These findings suggest that future medical chatbots should move beyond one-size-fits-all communication and adapt to individual users' situations, preferences, and information needs. Evaluation protocols should likewise assess not only whether chatbot responses are accurate and comprehensive but also whether they are actionable, appropriately toned, and responsive to users' evolving needs over the course of a conversation.