Evaluating AI-based breastfeeding chatbots: quality, readability, and reliability analysis

Area of Science:

• Artificial Intelligence in Healthcare
• Maternal and Child Health
• Digital Health

Background:

• Expectant and breastfeeding mothers increasingly use online platforms for information.
• AI-based chatbots are becoming common sources for automated breastfeeding guidance.

Purpose of the Study:

• To evaluate and compare the performance of ChatGPT, Gemini, and Copilot in providing breastfeeding information.
• Assess quality, reliability, readability, and response similarity across three AI chatbots.

Main Methods:

• Two researchers assessed AI chatbot responses to 50 common breastfeeding questions.
• Evaluated using the EQIP, SMOG, SI, mDISCERN, and GQS scoring criteria.

Main Results:

• Statistically significant differences were found in chatbot performance (p < 0.05).
• Copilot led in EQIP, SMOG, and SI; Gemini excelled in mDISCERN and GQS.
• All chatbots provided high-quality, reliable information, but readability required a university level.

Conclusions:

• AI chatbots offer reliable breastfeeding answers, though comprehension may be an issue.
• While more dependable than other online sources, AI accuracy and usability require further investigation.
• Advanced AI integration in healthcare necessitates continued research.