Not the Models You Are Looking For: Traditional ML Outperforms LLMs in Clinical Prediction Tasks.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Clinical Prediction Modeling

Background:

• Electronic Health Records (EHRs) contain vast data for clinical prediction.
• Large Language Models (LLMs) are emerging AI tools with potential in healthcare.
• Traditional Machine Learning (ML) is established for predictive tasks in medicine.

Purpose of the Study:

• To compare the efficacy of LLMs (GPT-3.5, GPT-4) against traditional ML for clinical prediction using EHR data.
• To assess LLM performance, calibration, fairness, and privacy robustness.
• To investigate the impact of in-context learning on LLM performance.

Main Methods:

• Evaluated GPT-3.5, GPT-4, and gradient-boosting trees (ML) on EHR data (VUMC, MIMIC IV).
• Measured predictive performance using AUROC and calibration with Brier Score.
• Assessed fairness using equalized odds and statistical parity; evaluated privacy resilience by generalizing demographic variables.

Main Results:

• Traditional ML significantly outperformed both GPT-3.5 and GPT-4 in predictive performance and calibration.
• ML models demonstrated greater robustness to privacy-preserving data generalization.
• GPT-4 exhibited better fairness but with compromised performance; LLMs showed poorer calibration than ML.

Conclusions:

• Current LLMs are less effective and robust than traditional ML for clinical prediction tasks with EHR data.
• LLMs show promise but require further development to match ML capabilities in this domain.
• Ongoing advancements in LLMs suggest potential for future clinical applications.