Large language models for frontline healthcare support in low-resource settings

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Global Health

Background:

• Existing benchmarks for large language models (LLMs) in medicine do not adequately represent low-resource health systems.
• There is a need to evaluate LLM performance in real-world clinical settings, especially in multilingual contexts.

Purpose of the Study:

• To develop a novel dataset of clinical questions from a low-resource setting in Rwanda.
• To compare the performance of five leading LLMs against local clinicians in answering these questions.
• To assess the cost-effectiveness and potential of LLMs in enhancing frontline healthcare.

Main Methods:

• A dataset of 5,609 clinical questions was created from 101 community health workers in Rwanda.
• Five LLMs (Gemini-2, GPT-4o, o3-mini, Deepseek R1, Meditron-70B) were evaluated.
• A subset of 524 question-answer pairs was scored by experts on 11 metrics using a 5-point Likert scale.
• LLM responses were compared to those of local clinicians.

Main Results:

• Gemini-2 and GPT-4o achieved the highest mean scores (4.49 and 4.48/5, respectively).
• All evaluated LLMs significantly outperformed local clinicians across all metrics (P < 0.001).
• LLMs demonstrated superiority even when communicating in Kinyarwanda and were over 500 times cheaper per response.

Conclusions:

• LLMs have significant potential to improve the quality of frontline care in low-resource, multilingual health systems.
• LLMs offer a cost-effective alternative to traditional healthcare provision in underserved regions.
• Further research should explore the integration of LLMs into existing health infrastructures.