LLM-Based Medical Document Evaluation: Integrating Human Expert Insights.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Large Language Models (LLMs) offer potential for automating medical document generation.
• Ensuring the reliability of LLM-generated medical documents requires significant expert input, hindering clinical adoption.
• Existing evaluation methods may not fully capture the nuances of LLM performance in clinical contexts.

Purpose of the Study:

• To develop and validate an LLM-based evaluation framework for medical document generation.
• To compare the effectiveness of different Chain of Thought (CoT) strategies within the LLM evaluation framework.
• To assess the alignment of the LLM-based evaluation with expert human judgment.

Main Methods:

• Developed a progressive LLM evaluation framework incorporating three Chain of Thought (CoT) strategies: Qualitative, Quantitative-qualitative, and Insight-integrated.
• Applied the framework to evaluate 33 LLM-generated Emergency Department records across five key criteria.
• Correlated the LLM-based evaluation results with expert evaluations using statistical analysis.

Main Results:

• The Insight-integrated CoT strategy demonstrated a strong correlation with expert evaluations (r = 0.680, p < .001).
• This approach outperformed the Qualitative (r = 0.524) and Quantitative-qualitative (r = 0.630) strategies in aligning with expert assessments.
• The framework effectively captured nuanced evaluation patterns while maintaining efficiency.

Conclusions:

• LLM-based evaluation frameworks, particularly the Insight-integrated approach, can effectively align with expert assessments for medical documentation.
• These frameworks show potential as efficient tools for validating the reliability of LLM-generated clinical records.
• Further development and application of such frameworks can facilitate the safe integration of LLMs into clinical workflows.