Evaluating a retrieval-augmented pregnancy chatbot: a comprehensibility-accuracy-readability study of the DIAN AI...

Patient education materials (PEMs) often exceed common health literacy levels. Retrieval-augmented conversational AI may deliver interactive, evidence-grounded explanations tailored to user needs. We evaluated DIAN, a RAG-enabled pregnancy chatbot grounded in the NHS Pregnancy Book, using a comprehensibility-accuracy-readability (CAR) framework to compare perceptions between women and clinicians across key perinatal domains.

We conducted a cross-sectional evaluation with standardized prompts and blinded scoring. Participants were 119 women (18-55 years) and 29 clinicians. After brief CAR training and calibration, all evaluators independently rated the same DIAN responses on 4-point Likert scales across postpartum care, pregnancy health and complications, diet and nutrition, and mental and emotional wellbeing. Between-group differences were tested using the Mann-Whitney U test with Bonferroni adjustment across domains per outcome; effect sizes were summarized with r = |Z|/√N and Cliff's delta. Inter-rater reliability was not estimated, given the independent-rater design.

Differences concentrated in postpartum care. Comprehensibility favored women (U = 1206.50, Z = -2.524, p = 0.012; r = 0.207; Δ = 0.301). Accuracy also favored women (U = 1239.00, Z = -2.370, p = 0.018; r = 0.195; Δ = 0.282). Readability favored clinicians (U = 1181.50, Z = -2.639, p = 0.008; r = 0.217; Δ = 0.315). Other domains showed no significant between-group differences after correction. Radar visualizations mirrored these patterns, with women showing larger comprehensibility/accuracy profiles and clinicians showing larger readability profiles in postpartum care.

Grounded in an authoritative national guide, DIAN achieved broadly comparable CAR perceptions across groups, with clinically relevant divergence limited to postpartum care. Women perceived higher comprehensibility and accuracy, while clinicians judged language more readable, suggesting a gap between experiential clarity and professional textual ease. Targeted postpartum refinement, lexical simplification, role-tailored summaries, and actionable checklists may align perceptions without compromising fidelity. More broadly, RAG-grounded chatbots can support equitable digital health education when content is vetted, updated, and evaluated with stakeholder-centered metrics. Future work should examine free-form interactions, longitudinal behavioral outcomes, and ethical safeguards (scope-of-use messaging, escalation pathways, and bias audits).