Evaluation Framework of Large Language Models in Medical Documentation: Development and Usability Study.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Large language models (LLMs) present opportunities for healthcare documentation but face challenges in accuracy, reliability, and standardization.
• Concerns exist regarding the clinical application of LLM-generated medical records due to quality assurance gaps.

Purpose of the Study:

• To develop and validate an evaluation framework for assessing the accuracy and clinical applicability of LLM-generated emergency department (ED) records.
• To enhance the integration of artificial intelligence in healthcare documentation through a robust evaluation system.

Main Methods:

• A competitive event, Healthcare Prompt-a-thon, involved 52 participants generating 33 ED records using HyperCLOVA X.
• A dual evaluation approach was used: clinical assessment by 4 medical professionals (Likert scale) and quantitative error analysis (7 error types).
• Statistical methods, including Pearson correlation and intraclass correlation coefficients (ICC), were employed for reliability and agreement assessment.

Main Results:

• Clinical evaluation showed strong interrater reliability (ICC 0.653-0.887) and test-retest reliability (Pearson r=0.776).
• Invalid generation errors were most common (35.38%), while structural errors most negatively impacted clinical scores (Pearson r=-0.654).
• A significant negative correlation (Pearson r=-0.633) was found between quantitative errors and clinical evaluation scores, indicating lower acceptability with higher errors.

Conclusions:

• The proposed evaluation framework is reliable and clinically acceptable for assessing LLM-generated ED records.
• The framework can mitigate clinical burdens and promote responsible AI integration in healthcare.
• This research offers a promising direction for future AI applications in medical documentation.