Performance of Large Language Models vs Conventional Machine Learning for Predicting Clinical Outcomes With Limited...

Area of Science:

• Clinical informatics
• Artificial intelligence in healthcare
• Predictive modeling

Background:

• Machine learning (ML) models typically require extensive patient data (hundreds to thousands of samples) for accurate clinical outcome prediction.
• Reducing data requirements to tens of patients would unlock applications in clinical trials and rare disease support.
• Large language models (LLMs) show promise in outperforming conventional ML on small, tabular datasets.

Purpose of the Study:

• To evaluate the performance of LLMs against conventional ML algorithms for predicting clinical outcomes.
• To assess the advantage of LLMs when training data is limited.

Main Methods:

• Compared two LLMs (one proprietary, one open-source) with conventional ML classification algorithms.
• Utilized three distinct, recently published clinical datasets (sepsis, gastric cancer, acute leukemia).
• Ensured datasets were published after LLM knowledge cutoffs and sampled to vary training set sizes.

Main Results:

• LLMs demonstrated superior performance compared to conventional ML for training datasets under 50 patients.
• Performance was measured using metrics including ROC AUC, F1-score, average precision, and balanced accuracy.
• The advantage of LLMs was attributed to their ability to leverage contextual information.

Conclusions:

• LLMs show significant potential for clinical outcome prediction with limited patient data (tens of patients).
• These findings may enable novel clinical use cases, such as optimizing clinical trial design and supporting rare disease diagnosis.
• Further optimization of LLM-based ML approaches is warranted.