Enhancing predictive modeling for respiratory support with LLM-driven guideline adherence

Optimal respiratory support selection between high-flow nasal cannula (HFNC) and noninvasive ventilation (NIV) for intensive care units (ICU) patients at risk of invasive mechanical ventilation (IMV) remains unclear, particularly in cases not represented in prior clinical trials. We previously developed RepFlow-CFR, a deep counterfactual model estimating individualized treatment effects (ITE) of HFNC versus NIV. However, interpretability and guideline alignment remain challenges for clinical adoption. This study describes the development and integration of a clinical guideline-driven LLM to enhance deep counterfactual model recommendations for NIV versus HFNC in patients at high-risk for invasive mechanical ventilation.

We enhanced RepFlow-CFR by incorporating a large language model (LLM, Claude 3.5 Sonnet) to enforce clinical guideline adherence and generate explainable treatment recommendations. The LLM was configured in a HIPAA-compliant AWS environment and prompted using structured patient data, clinical notes, and formal guideline criteria. Recommendations from RepFlow-CFR and LLM were compared to actual treatment decisions to assess concordance. We evaluated IMV and mortality/hospice rates across concordant and discordant groups. Additionally, we conducted a structured chart review of 20 cases to assess the clinical validity and safety of LLM-driven recommendations.

Among 1,261 ICU encounters, treatments concordant with LLM-enhanced recommendations were associated with lower IMV rates. For the HFNC recommendation, IMV occurred in 46/188 (24.47%) when care was concordant versus 9/17(52.94%) when discordant, corresponding to a 97.33% relative risk increase when discordant. Concordance was also associated with reduced mortality or hospice discharge (odds ratio 0.670, p = 0.046). In a 20-case chart review, 19/20 (95%) LLM recommendations aligned with clinical guidelines and physicians agreed with 13/20 (65%) final recommendations. Errors were noted in 11/20 cases, most rated low or moderate risk; 2/20 were judged as potentially causing severe harm.

Integrating LLMs for guideline enforcement improves the interpretability and clinical alignment of counterfactual models in respiratory support decision-making. This hybrid framework not only enhances concordance with real-world practice but may also improve patient outcomes. Future work will refine contraindication detection and expand validation to prospective clinical trials.