Developing an artificial intelligence-powered question-and-answer chatbot with English-Spanish capabilities for new...

Generative AI chatbots are revolutionizing health education by making complex information more accessible to the public. However, their use presents risks, including bias, hallucinations, ethical concerns, and misinformation, which are particularly critical in health contexts where incorrect guidance can have serious consequences. Ensuring safety, accuracy, and reliability is essential, especially in maternal and infant health.

We developed a multilingual chatbot, called Rosie, that employs a 3-stage AI pipeline, including a retriever, re-ranker, and generative model, to deliver efficient and relevant responses. To evaluate Rosie, we conducted a randomized controlled trial with pregnant and postpartum women (aged 14+ years, with infants under 6 months) from 49 US states (ClinicalTrials.gov ID NCT06053515). To analyze user interaction with Rosie, we examined 30 188 questions submitted by 197 users from October 2023 to April 2025. Six months after enrollment, a subset of Rosie participants were invited to complete a midpoint satisfaction survey assessing the chatbot's response quality, clarity, usefulness, and feature engagement. Of 105 eligible participants, 84 completed the survey (80% completion rate) between November 2024 and June 2025.

Users asked an average of 2.73 questions per day, with increased activity on weekdays and evenings, peaking at 10 p.m. The most highly rated topics included infant feeding, developmental milestones, symptoms (eg, fever), and hospital/birth preparations. Midpoint feedback was positive: 86% rated the answers as high quality, 91% found them useful, 95% found them easy to understand, and 84% expressed satisfaction.

In the initial phase, Rosie users reported technical issues and less satisfactory responses. However, integrating a retrieval-augmented generation system, expanding Rosie's knowledge base, and adding more interactive features led to sustained increases in positive ratings and consistently high user satisfaction. The Spanish-language expansion, enabled by a multilingual pipeline and advanced translation models, directly addressed pilot study feedback and further broadened Rosie's accessibility. Rosie offers a broad approach, supporting users through pregnancy, childbirth, postpartum, and infant care during the first year.

Chatbots like Rosie have the potential to transform health information delivery by providing a scalable, personalized, and cost-efficient solution.