Clinical Assessment of Fine-Tuned Open-Source LLMs in Cardiology: From Progress Notes to Discharge Summary

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Automating discharge summaries in specialized fields like cardiology is challenging due to data scarcity and complex terminology.
• Existing methods struggle with the accuracy and completeness required for clinical documentation.

Purpose of the Study:

• To develop and evaluate an automated system for generating accurate cardiology discharge summaries.
• To leverage synthetic data generation and fine-tuned large language models (LLMs) to overcome data limitations.

Main Methods:

• A hybrid dataset was created combining real and synthetically generated cardiology discharge summaries.
• The Llama3.1-8B large language model was fine-tuned on this dataset.
• Synthetic data quality was ensured using a T5-based binary classifier.

Main Results:

• The fine-tuned Llama3.1-8B model achieved superior performance in generating discharge summaries, validated by ROUGE, BLEU, and BERTScore metrics.
• Expert cardiologists confirmed the generated summaries were clinically coherent, complete, and accurate.
• The AI-assisted approach demonstrated high accuracy in capturing patient conditions and treatment details.

Conclusions:

• Fine-tuned open-source LLMs are feasible for specialized clinical documentation tasks.
• A validated framework for synthetic medical data augmentation in low-resource settings was established.
• AI-assisted clinical documentation offers a scalable solution to reduce administrative burden while maintaining patient care standards.