Large language models versus classical machine learning performance in COVID-19 mortality prediction using...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Computational Biology

Background:

• Predictive modeling for COVID-19 mortality is crucial for patient management and resource allocation.
• Classical machine learning models (CMLs) and large language models (LLMs) are increasingly used in healthcare analytics.

Purpose of the Study:

• To compare the performance of CMLs and LLMs in predicting COVID-19 mortality using high-dimensional tabular patient data.
• To evaluate the effectiveness of zero-shot and fine-tuned LLMs against established CML models.

Main Methods:

• Evaluated seven CML models (XGBoost, Random Forest) and eight LLMs (GPT-4, Mistral-7b) on 9,134 patient records.
• LLMs performed zero-shot classification on text-converted structured data; Mistral-7b was fine-tuned using QLoRA.

Main Results:

• XGBoost and Random Forest achieved high F1 scores (0.87 and 0.83).
• Zero-shot LLMs had moderate performance (GPT-4 F1: 0.43).
• Fine-tuned Mistral-7b significantly improved recall (1% to 79%) and achieved an F1 score of 0.74.

Conclusions:

• CMLs currently outperform LLMs for COVID-19 mortality prediction using high-dimensional structured data.
• Fine-tuning enhances LLM performance, showing potential for future medical predictive modeling.
• Both approaches offer valuable insights, but CMLs are preferred for current structured data tasks.