Expert evaluation of large language models for clinical dialogue summarization

Area of Science:

• Artificial Intelligence
• Clinical Informatics
• Natural Language Processing

Background:

• Automated summarization of clinical dialogues is crucial for efficient healthcare documentation.
• Evaluating the performance of large language models (LLMs) in this domain requires robust metrics and human oversight.

Purpose of the Study:

• To assess and compare the performance of various LLMs in summarizing clinical dialogues.
• To evaluate the reliability of computational metrics (ROUGE, UniEval) against expert human evaluation for clinical summary generation.

Main Methods:

• Exploratory evaluation of five LLMs, including general and fine-tuned models, and ChatGPT.
• Assessment using ROUGE and UniEval metrics.
• Expert clinician evaluation comparing model-generated summaries against a human-generated gold standard.

Main Results:

• ChatGPT demonstrated the highest scores in UniEval and clinician evaluations for coherence, consistency, fluency, and overall clinical utility.
• A fine-tuned transformer model performed best on ROUGE metrics, while ChatGPT performed lowest.
• UniEval showed strong correlation with human ratings, unlike ROUGE, suggesting its greater reliability for clinical summaries.

Conclusions:

• ChatGPT's performance in summarizing clinical dialogues approaches human quality.
• UniEval is a more reliable metric than ROUGE for evaluating automated clinical summary generation.
• LLMs offer a potential solution for automating clinical dialogue summarization, but privacy and data access remain significant challenges.