Related Experiment Video
Updated: Mar 22, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Feasibility of a Large Language Model Chatbot to Support Parental Understanding in the PICU
R Brandon Hunter1,2, Satid Thammasitboon1,2, Sreya Rahman1
1Department of Pediatrics, Baylor College of Medicine, Houston, TX.
Objectives:
To evaluate the feasibility of a large language model (LLM)-based chatbot for answering parental questions in the PICU and inform design of a randomized controlled trial (RCT).
Design:
Prospective single-arm feasibility study conducted from August 2024 to December 2024.
Setting:
Quaternary PICU.
Subjects:
Fourteen parents of children admitted to the PICU.
Interventions:
Parents engaged in 10-minute sessions with a HIPAA-compliant GPT-4o- (Generative Pretrained Transformer 4o, OpenAI, San Francisco, CA) based chatbot prompted with patient-specific electronic health record (EHR) data.
Measurements And Main Results:
Feasibility was assessed through four criteria: parental engagement and satisfaction, provider perceptions, accuracy and safety, and recruitment. Of 16 eligible parents, 14 enrolled and completed all procedures (87.5% recruitment rate). Parents asked a median of six questions (range, 3-13) with 96% positive real-time satisfaction ratings. Post-interaction surveys demonstrated high perceived value (median, 5.0/6.0 across all domains; Net Promoter Score [NPS] +57). Of 1225 chatbot-generated sentences evaluated, 99.3% were accurate with all eight errors classified as minor (inter-rater reliability: Gwet's AC2, a chance-corrected inter-rater agreement coefficient, = 0.98; 95% CI, 0.97-0.99). Healthcare providers rated response quality highly (median, 5.0/6.0), although physicians expressed greater comfort with bedside use of the tool than nurses (5.0 vs. 4.0; p = 0.004). Sample size calculations using NPS as the primary endpoint suggest enrolling 135 participants would provide adequate power for a future RCT.
Conclusions:
An EHR-informed LLM chatbot demonstrated high parental engagement and satisfaction, positive provider perception, and high accuracy and safety, supporting progression to a RCT.
