Applying Large Language Models for Surgical Case Length Prediction

Area of Science:

• Artificial Intelligence in Healthcare
• Surgical Operations Management
• Clinical Informatics

Background:

• Accurate prediction of surgical case duration is crucial for efficient operating room (OR) management.
• Inefficient scheduling leads to decreased patient and surgeon satisfaction and significant financial losses.

Purpose of the Study:

• To assess the feasibility and accuracy of large language models (LLMs) in predicting surgical case length.
• To compare LLM performance against existing surgical case duration estimation methods using unstructured clinical data.

Main Methods:

• Retrospective analysis of 125,493 elective surgical cases from 2017-2023.
• Eleven LLMs, including GPT-4, GPT-3.5, Mistral, Llama-3, and Phi-3, were evaluated, with fine-tuned variants of GPT-4 and GPT-3.5.
• Prediction accuracy was based on mean absolute error (MAE) and percentage of predictions within 20% of actual duration.

Main Results:

• Fine-tuned GPT-4 achieved the best performance (MAE 47.64 min, R2 0.61), comparable to current OR scheduling (MAE 49.34 min, R2 0.63).
• Fine-tuned LLMs (GPT-4 and GPT-3.5) significantly outperformed current methods in prediction accuracy (46.12% and 46.08% vs 40.92%, P < .001).
• Fine-tuned GPT-4 demonstrated strong performance in external validation (MAE 48.66 min, accuracy 46.0%).

Conclusions:

• Fine-tuned LLMs can predict surgical case length with accuracy comparable to or exceeding current institutional methods.
• LLMs show potential for enhancing operating room efficiency through improved case length prediction.
• Utilizing existing clinical documentation with LLMs offers a novel approach to surgical scheduling optimization.