Clinical Large Language Model Evaluation by Expert Review (CLEVER): Framework Development and Validation | JoVE Visualize

Area of Science:

Artificial Intelligence in Medicine
Natural Language Processing
Clinical Informatics

Background:

Evaluating large language models (LLMs) is challenging due to data contamination and the gap between benchmark tasks and clinical practice.
Existing LLM evaluation methods like public benchmarks and LLM-as-a-judge are limited by data issues and self-preference bias.
There is a critical need for robust evaluation frameworks that reflect real-world clinical utility.

Purpose of the Study:

To introduce CLEVER (Clinical Large Language Model Evaluation-Expert Review), a novel methodology for evaluating LLMs in healthcare.
To conduct a blind, randomized, preference-based evaluation of LLMs using practicing medical doctors.
To compare the performance of a general-purpose LLM against healthcare-specific LLMs on clinical tasks.

Main Methods:

The CLEVER methodology was employed to compare GPT-4o with two healthcare-specific LLMs (8B and 70B parameters).
Evaluations were performed on three distinct clinical tasks: text summarization, information extraction, and question answering.
Practicing medical doctors provided preference-based evaluations focusing on factuality, clinical relevance, and conciseness.

Main Results:

Medical doctors preferred the smaller, healthcare-specific LLM over GPT-4o in 45% to 92% of cases across key clinical dimensions.
The healthcare-specific LLM demonstrated superior performance in factuality, clinical relevance, and conciseness compared to GPT-4o.
Performance was comparable in open-ended medical question answering, indicating specialized LLMs excel in context-dependent tasks.

Conclusions:

Healthcare-specific LLMs can outperform larger, general-purpose LLMs in tasks requiring clinical context understanding.
The CLEVER methodology provides a valid and reliable approach for evaluating clinical LLMs, confirmed by interannotator agreement and correlation analyses.
This study underscores the importance of specialized LLMs and expert review for advancing AI in medicine.