Evaluation of Large Language Models for Peer Review in Transplantation Research: Algorithm Validation Study.

Area of Science:

• Medical research
• Artificial intelligence in science

Background:

• Peer review is crucial for research quality but faces challenges from reviewer fatigue and bias.
• The increasing volume of scientific publications necessitates exploring AI solutions like large language models (LLMs) for peer review support.

Purpose of the Study:

• To compare the performance of five open-source LLMs in reviewing organ transplantation papers.
• To evaluate the impact of author affiliations on LLM review outcomes.
• To examine the effectiveness of different prompt engineering strategies (zero-shot, few-shot, ToT, RAG) on LLM review decisions.

Main Methods:

• Evaluated 200 transplantation papers using five LLMs (Llama 3.3, Mistral 7B, Gemma 2, DeepSeek r1-distill Qwen, Qwen 2.5).
• Tested four prompt engineering strategies (zero-shot, few-shot, ToT, RAG) across various temperature settings.
• Assessed LLM performance on quartile categorization, considering author affiliations (prestigious, less prestigious, none) to detect bias.

Main Results:

• Retrieval-Augmented Generation (RAG) with a temperature of 0.5 yielded the best performance (0.35 exact match accuracy).
• LLMs tended to assign papers to middle quartiles (2 and 3), avoiding extremes.
• No significant affiliation bias was detected across models, though some showed marginal bias (Gemma 2, Qwen 2.5).
• Mistral demonstrated the highest accuracy (0.35) with the lowest computational cost.

Conclusions:

• Current open-source LLMs are not sufficiently accurate to replace human peer reviewers.
• While LLMs show potential for fairness by reducing affiliation bias, their accuracy limitations necessitate human supervision.
• Mistral offered the best balance of accuracy and efficiency; RAG is a promising prompting strategy.