Improving accuracy of GPT-3/4 results on biomedical data using a retrieval-augmented language model

Area of Science:

• Artificial Intelligence
• Biomedical Research
• Natural Language Processing

Background:

• General-purpose large language models (LLMs) possess broad knowledge but may generate inaccuracies in specialized domains.
• Domain-specific LLMs enhance precision but require extensive resources for training.
• Retrieval-augmented generation (RAG) offers a potential solution to improve LLM performance in specific fields without intensive retraining.

Purpose of the Study:

• To evaluate the performance of a retrieval-augmented generation (RAG) model compared to general-purpose LLMs for answering questions in a specific biomedical research area.
• To assess accuracy, relevance, and readability of responses generated by different LLMs and a custom RAG model.

Main Methods:

• A custom RAG model and existing LLMs (OpenAI's GPT-3.5, GPT-4, Microsoft's Prometheus) were used to answer 19 questions on diffuse large B-cell lymphoma (DLBCL) biology and treatment.
• Eight independent reviewers evaluated the LLM responses on accuracy, relevance, and readability using a 3-point scale.
• Performance metrics were analyzed to compare the models.

Main Results:

• The RAG model achieved higher average scores for accuracy and relevance compared to the general-purpose LLMs.
• GPT-4 demonstrated comparable relevance to the RAG model but lower accuracy.
• GPT-4 and GPT-3.5 exhibited higher readability scores but also more instances of hallucinations (e.g., fabricated references, inaccurate clinical information).

Conclusions:

• An oncology research-focused RAG model can outperform general-purpose LLMs in accuracy and relevance for subject-specific queries.
• The RAG framework is adaptable for question-answering in various specialized domains.
• Further research is needed to explore the impact of different LLM architectures, RAG methodologies, and prompting strategies.