Evaluation over Generalist Large Language Models and Specialised Models for Clinical Risk Prediction

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Biomedical Data Science

Background:

• Large language models (LLMs) demonstrate potential in healthcare.
• LLMs face accuracy challenges in disease risk prediction, particularly for rare conditions.
• Evaluating LLM performance against established clinical prediction models is crucial.

Purpose of the Study:

• To assess the accuracy of standalone and collaborative LLMs (ChatGPT, DeepSeek) for disease risk prediction.
• To compare LLM performance against task-specific models for rare (cholangiocarcinoma) and common (myocardial infarction) diseases.
• To explore the potential of hybrid AI architectures in clinical settings.

Main Methods:

• Comparative analysis of standalone LLMs (ChatGPT, DeepSeek) and task-specific models.
• Evaluation of a collaborative prediction framework integrating LLMs and specialized models.
• Disease risk prediction tasks for cholangiocarcinoma and myocardial infarction.

Main Results:

• Standalone LLMs underperformed specialized models in disease risk prediction.
• Collaborative prediction frameworks significantly improved the performance of LLMs.
• Hybrid approaches demonstrated enhanced accuracy, especially for rare disease prediction.

Conclusions:

• LLMs alone have limitations for clinical disease risk prediction, especially for rare conditions.
• Hybrid AI architectures combining LLMs with specialized models offer a promising direction for clinical AI.
• Collaborative prediction strategies can leverage the complementary strengths of different AI approaches for improved healthcare outcomes.