Large Language Models for Maternal and Neonatal Health Care in Low- and Middle-Income Countries.

Area of Science:

• Artificial Intelligence in Healthcare
• Global Maternal and Neonatal Health
• Health Informatics

Background:

• Low- and middle-income countries (LMICs) face significant challenges in maternal and neonatal healthcare.
• Large language models (LLMs) offer potential for information dissemination and decision support.

Purpose of the Study:

• To investigate the efficacy of LLMs, specifically GPT-4o, in supporting maternal and neonatal healthcare in LMICs.
• To evaluate GPT-4o's accuracy in generating health information relevant to LMICs.

Main Methods:

• GPT-4o's ability to generate accurate answers was assessed across four domains: condition prevalence in medical cases, prevalence data for data-scarce regions, standardized medical exam questions, and subjective health queries.
• Publicly available data from 6 WHO regions and 204 countries were utilized.
• Repeated prompts were employed to ensure the statistical significance of the LLM's responses.

Main Results:

• GPT-4o generated medical cases that did not accurately reflect global condition prevalences, overrepresenting the Americas.
• The LLM exhibited bias, assigning lower rankings to countries with high infant mortality rates for subjective health topics.
• In 44% of instances, GPT-4o provided unavailable pregnancy statistics without acknowledging uncertainty, and 46.7% of its source citations were erroneous.
• While GPT-4o answered 79% of general pregnancy medical examination questions correctly, it made errors when regional health issues and symptoms were considered.

Conclusions:

• General-purpose LLMs like GPT-4o have significant limitations for guiding maternal and neonatal healthcare in LMICs.
• Findings underscore the need for fine-tuning LLMs with context-specific data to improve accuracy and reduce bias.
• Further research is required to develop tailored LLM solutions for LMIC healthcare contexts.