The foundational capabilities of large language models in predicting postoperative risks using clinical notes.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Data Analysis

Background:

• Clinical notes contain valuable perioperative patient data.
• Large language models (LLMs) present opportunities to leverage this data.
• Predicting postoperative risks is crucial for patient outcomes.

Purpose of the Study:

• To evaluate the performance of LLMs in predicting six postoperative risks.
• To compare different LLM fine-tuning strategies against traditional methods.
• To assess the potential of LLMs in perioperative care.

Main Methods:

• Utilized 84,875 preoperative notes and surgical cases (2018-2021).
• Compared pretrained LLMs with traditional word embeddings.
• Investigated self-supervised fine-tuning and label incorporation.
• Evaluated performance using Area Under the Receiver Operating Characteristic Curve (AUROC) and Area Under the Precision-Recall Curve (AUPRC).

Main Results:

• Pretrained LLMs significantly outperformed traditional word embeddings (AUROC +38.3%, AUPRC +33.2%).
• Self-supervised fine-tuning yielded further improvements (AUROC +3.2%, AUPRC +1.5%).
• Label incorporation enhanced performance (AUROC +1.8%, AUPRC +2%).
• Unified foundation models achieved the highest performance (AUROC +3.6%, AUPRC +2.6% over self-supervision).

Conclusions:

• LLMs demonstrate strong capabilities in predicting postoperative risks from clinical notes.
• Advanced fine-tuning strategies, particularly unified foundation models, optimize LLM performance.
• LLM integration holds significant potential for improving perioperative care and patient safety.