Mixed-methods evaluation of clinician experiences and adoption patterns of an EHR-integrated generative AI-based...

To quantify the patterns of uptake of a large language model (LLM)-based clinical decision support system across private primary health facilities in Kenya (operated by Penda Health) and explore factors influencing clinician uptake and overall experience.

A mixed-methods study combining quantitative analysis of clinical decision support system (CDSS) metadata from all consultations that took place between 1 February and 1 October 2024, augmented by qualitative data from 42 staff members (26 clinical officers, 10 facility managers, 4 nurses, 1 quality assurance manager and 1 business analyst). Data collection included journey mapping interviews (n=7), user-experience interviews (n=25), focus groups (n=2) and system utilisation metrics. Quantitative data were summarised using descriptive statistics, and qualitative data were analysed using thematic analysis (drawing on established theories of technology adoption and change management).

In total, there were 258 106 clinical episodes across all Penda Health facilities over the 8-month observation period, of which 56 050 (21.7%) were augmented by use of the 'artificial intelligence (AI) Consult'. 'AI Consult' use aggregated across the 16 facilities increased from 4% to 47% over 8 months. Clinicians provided feedback on their experience using the AI Consult in 31% of the clinical episodes; nearly all the feedback provided (99.5%) was positive. The qualitative investigation identified five key themes associated with clinicians' experiences with the AI Consult tool: (1) there are several value propositions to an 'AI Consult' style tool, (2) clinicians' confidence in the AI Consult grew with time, (3) clinicians' application of the 'AI Consult' is influenced by case complexity, (4) responses from the AI Consult are largely believed and valued by clinicians but several improvements are recommended and (5) clinicians find the 'AI Consult' easy to use but identified several pain points that warrant attention.

Successful generative AI/LLM-enhanced CDSS implementation in resource-constrained settings requires: (1) robust technological infrastructure, (2) localisation to reflect clinical guidelines, (3) structured change management with clinical champions and (4) seamless workflow integration. Future product development exercises should specifically consider alternatives to active solicitation of CDSS input, as it is liable to overconfidence-related underutilisation.