Enhancing perinatal health patient information through ChatGPT - An accuracy study

Area of Science:

• Perinatal Health
• Artificial Intelligence in Healthcare
• Medical Information Accuracy

Background:

• Growing interest in AI for healthcare applications.
• Need to assess the accuracy of AI tools like ChatGPT for critical health information.
• Limited research on AI accuracy in perinatal health domains.

Purpose of the Study:

• Evaluate ChatGPT's accuracy as an information source for pregnant women and maternity care workers.
• Assess recommendations on "nutrition" and "red flags" during pregnancy.
• Investigate the impact of language and question framing on accuracy.

Main Methods:

• ChatGPT-generated recommendations assessed using a 5-point Likert scale by eight raters.
• Evaluated ten indicators per topic across four languages: French, English, German, and Dutch.
• Calculated accuracy scores and inter-rater agreement for each topic and language.

Main Results:

• Excellent median accuracy scores (5.0; IQR 4-5) for both nutrition and red flags, irrespective of language.
• Overall accuracy: 83-89% for nutrition, 96-98% for red flags.
• Good to excellent inter-rater agreement observed for both topics.

Conclusions:

• ChatGPT generates accurate recommendations for pregnancy nutrition and red flags.
• Users must be aware of ChatGPT's limitations, including formulation, language, and context-dependent inconsistencies.
• Further research needed to understand AI's role and limitations in perinatal care.