Let large language models judge each other: multi-agent peer-reviewed reasoning for medical question answering

Area of Science:

• Artificial Intelligence
• Biomedical Informatics
• Natural Language Processing

Background:

• Large language models (LLMs) show promise in medical question answering (MedQA).
• Current LLM approaches may lack accuracy, interpretability, and robustness.
• Enhancing LLM performance in MedQA is crucial for reliable biomedical AI.

Purpose of the Study:

• To improve the accuracy, interpretability, and robustness of LLMs in MedQA.
• To introduce a novel multi-agent peer-reviewed reasoning framework for MedQA.

Main Methods:

• A multi-agent system where LLM agents generate chain-of-thought (CoT) reasoning and candidate answers.
• Agents independently peer-review each other's reasoning for factual and logical correctness.
• The highest-rated reasoning chain is selected for the final answer.
• Experiments utilized 5 LLMs on 3 benchmark datasets (HeadQA, MedQA-USMLE, PubMedQA).

Main Results:

• Peer-reviewed reasoning consistently outperformed single-model CoT and majority voting baselines.
• The best model combination achieved an average accuracy of 0.820, surpassing single models (0.777) and voting ensembles (0.789).
• The method demonstrated effective scaling with more models and reliable quality assessment of reasoning chains.

Conclusions:

• The multi-agent peer-reviewed reasoning method enhances LLM performance in MedQA by integrating solver and evaluator roles.
• This approach prioritizes reasoning quality, leading to improved accuracy, interpretability, and robustness.
• The framework offers a path toward more trustworthy biomedical AI systems.