Generative AI-enabled clinical decision support system in primary care: a pragmatic, cluster-randomized trial

Area of Science:

• Clinical Informatics
• Artificial Intelligence in Healthcare
• Global Health

Background:

• Limited evidence exists on large language model (LLM) performance in low-resource clinical settings.
• Real-world data on LLM integration into primary care is crucial for assessing their utility.

Purpose of the Study:

• To evaluate the impact of LLM assistance on patient treatment failure in Kenyan primary care.
• To assess the safety and efficacy of LLMs in a pragmatic, low-resource clinical environment.

Main Methods:

• A pragmatic, cluster-randomized trial was conducted in 16 primary care facilities in Kenya.
• Clinical officers used electronic medical records with or without LLM assistance for 9,691 enrolled patients.
• The primary outcome was an expert-adjudicated composite of treatment failure events within 14 days.

Main Results:

• Treatment failure occurred in 2.2% of patients in the LLM-assisted arm and 2.0% in the control arm (aOR 0.77, 95% CI 0.55-1.08, P=0.13).
• No significant difference in treatment failure rates was observed between the LLM-assisted and control groups.
• No serious adverse events were attributed to the LLM intervention, indicating a favorable safety profile.

Conclusions:

• LLM assistance in this low-resource primary care setting was safe but did not significantly reduce 14-day treatment failure.
• The study suggests that any potential benefits of LLMs in such settings may be modest.
• Further research may be needed to identify specific clinical applications where LLMs offer demonstrable advantages.