MedPromptEval: A Comprehensive Framework for Systematic Evaluation of Clinical Question Answering Systems

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Clinical deployment of large language models (LLMs) faces significant hurdles, including inconsistent performance and a lack of standardized evaluation methods.
• Existing evaluation approaches for clinical LLMs are insufficient for ensuring reliable performance in healthcare settings.

Purpose of the Study:

• To introduce MedPromptEval, a novel framework designed for the systematic evaluation of large language model (LLM) and prompt combinations in clinical contexts.
• To enable reproducible benchmarking and provide insights for optimizing prompt engineering and model selection for clinical question answering (QA) applications.

Main Methods:

• MedPromptEval automatically generates diverse prompt types and orchestrates response generation across multiple LLMs.
• Performance is quantified using metrics for factual accuracy, semantic relevance, entailment consistency, and linguistic appropriateness.
• The framework was demonstrated on clinical QA datasets (MedQuAD, PubMedQA, HealthCareMagic) in model comparison, prompt optimization, and configuration assessment modes.

Main Results:

• MedPromptEval facilitates systematic evaluation of LLM-prompt interactions across clinically relevant dimensions.
• Demonstrated utility in comparing models, optimizing prompt strategies, and assessing prompt-model configurations on public clinical QA datasets.
• The framework enables reproducible benchmarking for clinical LLM and QA applications.

Conclusions:

• MedPromptEval addresses critical challenges in the clinical deployment of LLMs by providing a standardized evaluation methodology.
• The framework advances the reliable and effective integration of language models in healthcare by offering insights for prompt engineering and model selection.