Identifying and timing patient outcomes in clinician notes using large language models

Area of Science:

• Medical informatics
• Clinical data science
• Artificial intelligence in healthcare

Background:

• Leveraging unstructured clinician notes for predictive modeling faces challenges in accurately identifying and timing patient outcomes.
• Large language models (LLMs) offer a potential solution for extracting temporal outcome data from clinical text.

Purpose of the Study:

• To apply LLMs for identifying and temporally localizing patient outcomes within clinician notes.
• To evaluate the enhancement of predictive modeling for conditions like sepsis using this contextual data.

Main Methods:

• Utilized the Medical Concept Annotation Tool (MedCAT) and two LLMs (Meta-Llama-3.1-8B, BioMistral 7B) on the MIMIC-III dataset.
• Manually validated LLM-identified outcomes, including sepsis, by a physician.
• Assessed the impact on sepsis prediction using downstream time series modeling.

Main Results:

• Meta-Llama-3.1-8B achieved a strong balance of coverage and precision in outcome identification.
• Physician validation confirmed varying model performance across different outcomes.
• Incorporating temporally localized outcomes improved sepsis prediction AUC from 0.77 to 0.84.

Conclusions:

• LLM-driven identification and timing of patient outcomes from unstructured notes are feasible.
• Contextual outcome data extracted by LLMs enhances predictive modeling capabilities.
• The TIMED-MIMIC dataset, containing 1697 localized outcomes, is released as a public resource.