Identifying and timing patient outcomes in clinician notes using large language models

Key challenges in leveraging unstructured clinician notes for predictive models include identifying and timing patient outcomes. To address these challenges we applied large language models (LLMs) to identify and temporally localize patient outcomes in clinician notes, and evaluated whether this contextual data enhances predictive modeling for conditions like sepsis.

We applied the Medical Concept Annotation Tool (MedCAT) and two LLMs, Meta-Llama-3.1-8B and BioMistral 7B, to clinician notes in the Medical Information Mart for Intensive Care version III to identify and time International Classification of Diseases-based patient outcomes. A physician manually validated the accuracy of the models' ability to identify and time outcomes including sepsis. Finally, downstream time series predictive modeling was used to assess the impact of unstructured clinician notes on sepsis prediction.

Meta-Llama-3.1-8B demonstrated the best balance of coverage and precision, reliably identifying outcomes while minimizing false positives. Our manual physician validation evaluated model performance, which varied between outcomes. When comparing derived sepsis timestamps to clinical sepsis onset within a 5-h window, Meta-Llama-3.1-8B showed the smallest median time difference. Finally, incorporating temporally localized outcomes into downstream sepsis prediction improved model performance, area under the receiver operating characteristic (AUC) of 0.84 (95% CI: 0.83-0.86) from 0.77 (95% CI: 0.76-0.79).

We demonstrate that LLM-driven identification and timing of patient outcomes from unstructured clinician notes is feasible. Contextual outcome identification unlocks the potential of unstructured clinician notes for predictive modeling. We provide TIMED-MIMIC, an automatically generated dataset encompassing 1697 temporally localized patient outcomes, as a publicly available resource.