GeneGPT: augmenting large language models with domain tools for improved access to biomedical information

Area of Science:

• Genomics
• Bioinformatics
• Artificial Intelligence

Background:

• Large language models (LLMs) face challenges with factual accuracy (hallucinations).
• Integrating LLMs with domain-specific tools like database utilities improves access to specialized knowledge.

Purpose of the Study:

• To develop GeneGPT, a novel method for LLMs to utilize National Center for Biotechnology Information (NCBI) Web APIs.
• To enhance LLM performance in answering genomics-related questions accurately.

Main Methods:

• Prompting Codex (a LLM) to use NCBI Web APIs via in-context learning.
• Employing an augmented decoding algorithm for detecting and executing API calls.
• Utilizing the GeneTuring benchmark and introducing the GeneHop dataset.

Main Results:

• GeneGPT achieved state-of-the-art performance on the GeneTuring benchmark (0.83 average score).
• GeneGPT significantly outperformed existing LLMs, including retrieval-augmented and biomedical models.
• API demonstrations proved more effective than documentation for in-context learning; GeneGPT generalized to multi-hop questions.

Conclusions:

• GeneGPT offers a robust solution for improving LLM accuracy in genomics.
• The method demonstrates effective API utilization and generalization capabilities.
• Further analysis provides insights for future LLM development in specialized domains.