Enhancing privacy-preserving deployable large language models for perioperative complication detection: a targeted...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Natural Language Processing
• Healthcare Informatics

Background:

• Perioperative complications pose a significant global health challenge, with manual detection methods exhibiting substantial under-reporting (27%) and misclassification rates.
• Deploying clinical Large Language Models (LLMs) is hindered by data privacy concerns, high computational costs, and suboptimal performance of locally deployable models.

Purpose of the Study:

• To develop and validate a framework using targeted prompt engineering and Low-Rank Adaptation (LoRA) fine-tuning to enhance the diagnostic capabilities of smaller, open-source LLMs for perioperative complication detection and severity grading.
• To evaluate the performance of optimized LLMs against human experts and assess their robustness to variations in clinical documentation complexity.

Main Methods:

• A dual-center validation framework was established to simultaneously identify and grade 22 distinct perioperative complication severities.
• Targeted prompt engineering, including Chain-of-Thought prompting, and LoRA fine-tuning were applied to smaller open-source LLMs.
• Performance was evaluated using F1 scores across different documentation length quartiles and compared between AI models and human experts.

Main Results:

• Optimized LLMs, particularly the 4B and 8B parameter models, demonstrated expert-level accuracy in identifying and grading perioperative complications, with the 8B model surpassing human expert performance (F1 > 0.70).
• The targeted strategy significantly improved model performance (ΔF1=0.256 for 4B model), with further gains from LoRA (ΔF1=0.103 for 4B model), raising the 4B model's micro-F1 to 0.64 on external validation.
• AI models exhibited superior robustness to documentation complexity, maintaining high F1 scores (F1 > 0.64) while human expert performance declined significantly (from 0.73 to 0.45).

Conclusions:

• Targeted prompt engineering combined with LoRA fine-tuning effectively transforms smaller open-source LLMs into high-performing clinical diagnostic tools for perioperative complications.
• These optimized small models offer a practical solution for resource-limited healthcare settings, enabling expert-level accuracy with local deployment and preserved data sovereignty.
• The study highlights a viable pathway for leveraging AI in clinical settings, addressing critical challenges in complication detection and management.