ClinicRealm: Re-evaluating large language models with conventional machine learning for non-generative clinical...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Machine Learning for Healthcare

Background:

• Large Language Models (LLMs) are increasingly used in healthcare, but their effectiveness for non-generative clinical prediction tasks remains under-evaluated.
• There is a common assumption that specialized models are superior to LLMs for clinical prediction, potentially leading to misuse and misunderstanding.
• The ClinicRealm benchmark was developed to systematically assess LLM performance in clinical settings.

Purpose of the Study:

• To systematically evaluate the performance of various Large Language Models (LLMs) against traditional methods for clinical prediction tasks.
• To compare LLM capabilities on both unstructured clinical notes and structured Electronic Health Records (EHR).
• To assess predictive performance, reasoning abilities, and fairness across different model types.

Main Methods:

• Evaluated 15 GPT-style LLMs, 5 BERT-style models, and 11 traditional methods using the ClinicRealm benchmark.
• Utilized unstructured clinical notes and structured Electronic Health Records (EHR) data for evaluation.
• Assessed models on predictive performance, reasoning, and fairness metrics.

Main Results:

• Leading zero-shot LLMs (e.g., DeepSeek-V3.1-Think, GPT-5) significantly outperformed fine-tuned BERT models on clinical notes.
• On structured EHR data, advanced LLMs demonstrated strong zero-shot capabilities, often surpassing conventional models in data-scarce scenarios.
• Leading open-source LLMs achieved performance comparable to or exceeding proprietary models.

Conclusions:

• Modern LLMs are highly competitive for clinical prediction tasks, challenging the assumption of their inferiority to specialized models.
• LLMs show significant promise for both unstructured and structured clinical data, particularly in data-limited situations.
• Health data scientists and developers should re-evaluate model selection strategies considering the advanced capabilities of contemporary LLMs.