RefAI: a GPT-powered retrieval-augmented generative tool for biomedical literature recommendation and summarization

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Biomedical professionals require precise literature recommendation and summarization.
• Current large language models (LLMs) like GPT have limitations in real-time search (fabricated papers) and summarization (poor source citation).

Purpose of the Study:

• Introduce RefAI, a novel retrieval-augmented generative tool.
• Synergize the strengths of LLMs while overcoming their inherent limitations for literature tasks.

Main Methods:

• Utilized PubMed for systematic literature retrieval.
• Employed a novel multivariable algorithm for article recommendation.
• Leveraged GPT-4 Turbo for summarization, comparing against ChatGPT-4, ScholarAI, and Gemini using 10 queries across 'cancer immunotherapy and target therapy' and 'LLMs in medicine'.

Main Results:

• RefAI significantly outperformed baseline tools across literature recommendation (relevance, quality) and summarization (accuracy, comprehensiveness, reference integration).
• Statistically significant improvements (P < .05) were observed in most evaluated dimensions.

Conclusions:

• RefAI addresses critical issues such as fabricated papers, inaccurate metadata, limited recommendations, and poor reference integration.
• Augmenting LLMs with external resources and a novel ranking algorithm enables high-quality literature recommendation and structured summarization.
• RefAI has the potential to meet the essential needs of biomedical professionals in managing and synthesizing scientific literature.