Benchmarking Large Language Models for MIMIC-IV Clinical Note Summarization

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) show promise for healthcare applications.
• Clinical note summarization effectiveness of LLMs is underexplored.
• Systematic comparisons of LLMs for this task are lacking.

Purpose of the Study:

• To benchmark 16 generative LLMs for clinical note summarization.
• To evaluate both extractive and abstractive summarization approaches.
• To assess performance, processing time, cost, and deployment feasibility.

Main Methods:

• Benchmarking 16 generative LLMs (OpenAI GPT, DeepSeek, Meta LLaMA, Google Gemma, Mistral Mixtral, Alibaba Qwen) on the MIMIC-IV-Note dataset.
• Implementing and evaluating extractive and abstractive summarization.
• Using lexical (ROUGE, BLEU, METEOR) and semantic (COMET, BERTScore) metrics.
• Assessing processing time, cost, and deployment feasibility.

Main Results:

• Gemma-3-27B demonstrated superior performance in extractive summarization.
• DeepSeek-R1-70B, Qwen-3-32B, and GPT-4o were top performers in abstractive summarization.
• Smaller models (LLaMa-3-8B, Gemma-2-9B) offered competitive results with improved efficiency.

Conclusions:

• Model selection for clinical note summarization involves trade-offs between performance, efficiency, and deployment context.
• LLM performance is not solely dependent on parameter size.
• Findings provide practical insights for integrating LLMs into healthcare workflows.