Synthesizing scientific literature with retrieval-augmented language models.

Area of Science:

• Artificial Intelligence
• Scientific Literature Synthesis
• Information Retrieval

Background:

• Scientific advancement relies on synthesizing vast amounts of research literature.
• Existing language models (LLMs) struggle with accurate scientific information retrieval and citation.

Purpose of the Study:

• To introduce OpenScholar, a retrieval-augmented language model designed for scientific literature synthesis.
• To evaluate OpenScholar's performance against state-of-the-art models on a novel benchmark for scientific literature search.

Main Methods:

• Developed OpenScholar, a specialized retrieval-augmented language model utilizing a data store of 45 million open-access papers.
• Created ScholarQABench, a large-scale, multi-domain benchmark with expert-written queries and answers across computer science, physics, neuroscience, and biomedicine.
• Implemented a self-feedback inference loop to enhance retrieval and synthesis capabilities.

Main Results:

• OpenScholar-8B outperformed GPT-4o and PaperQA2 in correctness on multi-paper synthesis tasks from ScholarQABench.
• OpenScholar demonstrated human-expert-level citation accuracy, unlike GPT-4o's high hallucination rate.
• OpenScholar-GPT-4o improved GPT-4o's correctness by 12%, with human experts preferring OpenScholar responses over expert-written ones 70% of the time.

Conclusions:

• OpenScholar significantly advances the capability of LLMs in synthesizing scientific literature accurately and with reliable citations.
• The developed benchmark, ScholarQABench, provides a robust platform for evaluating future scientific literature search tools.
• OpenScholar's architecture and components offer a pathway to improve existing LLMs for scientific applications.