Related Experiment Video
Updated: Jun 9, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Human-supervised, large language model-based clinical decision support aligned to national newborn protocols in
Teresiah Kuria1, Gideon Kamau2, Felisters Makokha3
1Centre for Research in Infectious Diseases, Mount Kenya University, Thika, Kenya.
Introduction:
Timely, protocol-adherent clinical decisions are crucial for reducing neonatal mortality in low-resource settings. Translating extensive national guidelines into bedside practice remains challenging.
Objective:
We developed and evaluated AIFYA, a human-supervised, large language model (LLM)-based clinical decision support system (CDSS) aligned with Kenya's national newborn care protocols.
Methods:
This prospective, mixed-methods, early-stage evaluation, guided by the DECIDE-AI framework, embedded AIFYA into routine workflows at two public health facilities (Level 5 and Level 4) in Bungoma County, Kenya, from September 2024 to June 2025. Primary outcomes were: (1) adoption, measured by cumulative neonatal cases managed; (2) training reach, assessed by credentialed healthcare workers (HCWs); and (3) guideline and citation concordance, evaluated through blinded review of 118 AI-generated recommendations by two neonatologists, with adjudication by a third. Secondary outcomes included protocol adherence and triage-to-decision time.
Results:
A total of 50 HCWs were trained, and 550 neonatal cases were managed over 10 months. Among surveyed HCWs (n = 33), 76% were female (mean age 32.1 years). Expert review found 75% of recommendations were correct and 15% partially correct, with strong inter-rater reliability (weighted Cohen's kappa 0.85; 95% CI 0.79-0.91) between reviewers. Citation accuracy was 96%. In 40 complex dosing scenarios, 75% of outputs were rated correct. The median triage-to-decision time was 23 min (IQR 18-31). Implementation was supported by an offline-first architecture and a facility-based coaching model, sustaining engagement despite staff turnover.
Conclusion:
A human-supervised, guideline-aligned AI CDSS can be feasibly implemented in routine neonatal care in low-resource settings, with high adoption and guideline concordance. Citation-linked recommendations enhance transparency and support clinical verification with most clinically appropriate recommendations. However, variability in complex scenarios highlights the need for ongoing refinement and strict clinical oversight. These findings support progression to control trials to evaluate clinical effectiveness and safety.
