A Multiagent Summarization and Auto-Evaluation Framework for Medical Text: Development and Evaluation Study

Although large language models (LLMs) show great promise in processing medical text, they are prone to generating incorrect information, commonly referred to as hallucinations. These inaccuracies present a significant risk for clinical applications where precision is critical. Additionally, relying on human experts to review LLM-generated content to ensure accuracy is costly and time-consuming, which sets a barrier against large-scale deployment of LLMs in health care settings.

The primary objective of this study was to develop an automatic artificial intelligence (AI) system capable of extracting structured information from unstructured medical data and using advanced reasoning techniques to support reliable clinical decision making. A key aspect of this objective is ensuring that the system incorporates self-verification mechanisms, enabling it to assess the accuracy and reliability of its own outputs. By integrating such mechanisms, we aim to enhance the system's robustness, reduce reliance on human intervention, and improve the overall trustworthiness of AI-driven medical summarization and evaluation.

The proposed framework comprises 2 layers: a summarization layer and an evaluation layer. The summarization layer uses Llama2-70B (Meta AI) and Mistral-7B (Mistral AI) models to generate concise summaries from unstructured medical data, focusing on tasks such as consumer health question summarization, biomedical answer summarization, and dialog summarization. The evaluation layer uses GPT-4-turbo (OpenAI) as a judge, leveraging pairwise comparison strategies and different prompt strategies to evaluate summaries across 4 dimensions: coherence, consistency, fluency, and relevance. To validate the framework, we compare the judgments generated by the LLM assistants in the evaluation layer with those provided by medical experts, offering valuable insights into the alignment and reliability of AI-driven evaluations within the medical domain. We also explore a way to handle disagreement among human experts and discuss our methodology in addressing diversity in human perspectives.

The study found variability in expert consensus, with average agreement rates of 19.2% among all experts and 54% among groups of 3 experts. GPT-4 (OpenAI) demonstrated alignment with expert judgments, achieving an average agreement rate of 83.06% with at least 1 expert and comparable performance in cross-validation tests. The enhanced guidance in prompt design (prompt-enhanced guidance) improved GPT-4's alignment with expert evaluations compared with a baseline prompt, highlighting the importance of effective prompt engineering in auto-evaluation of summarization tasks. We also evaluated open-source LLMs, including Llama-3.3 (Meta AI) and Mixtral-Large (Mistral AI), and a domain-specific LLM, OpenBioLLM (Aaditya Ura), for comparison as LLM judges.

This study highlights the potential of LLMs as reliable tools for unstructured medical data summarization and evaluation to reduce the dependency on human experts and also states the limitations. The proposed framework, multiagent summarization and auto-evaluation, demonstrates scalability and adaptability for clinical applications while addressing key challenges like hallucination and position bias.