Evaluating LLMs for Diagnosis Summarization.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Hospitalization generates extensive patient data in clinical notes.
• Efficient summarization of this information is crucial for healthcare professionals.
• Automating diagnosis summarization can improve clinical workflow.

Purpose of the Study:

• To assess the efficacy of six large language models (LLMs) in automating diagnosis summarization.
• To develop and validate an automatic metric for evaluating LLM performance in this task.
• To compare LLM performance against human expert assessments.

Main Methods:

• Proposed a methodology to evaluate six LLMs for diagnosis summarization in discharge summaries.
• Developed an automatic metric based on LLMs, correlated with human assessments.
• Evaluated models using F1-Score and compared results with healthcare specialists.

Main Results:

• LLMs demonstrate potential in automating diagnosis summarization.
• An LLM-based metric showed high correlation with human assessments.
• Significant room for improvement exists in LLMs' medical knowledge and diagnostic capabilities.

Conclusions:

• Automated diagnosis summarization using LLMs is a promising area.
• Current LLMs require further refinement for clinical application.
• The developed methodology and metric can guide future LLM development in healthcare.