Benchmarking large language models for genomic knowledge with GeneTuring.

Area of Science:

• Genomics
• Bioinformatics
• Artificial Intelligence

Background:

• Large language models (LLMs) show potential in biomedical research.
• The effectiveness of LLMs for genomic inquiry is not well-established.
• A need exists for standardized evaluation of LLMs in genomics.

Purpose of the Study:

• To develop a comprehensive benchmark for evaluating LLMs in genomics.
• To assess the performance of various LLM configurations on genomics tasks.
• To identify optimal strategies for integrating LLMs into genomic research.

Main Methods:

• Creation of GeneTuring, a benchmark with 16 genomics tasks and 1600 curated questions.
• Manual evaluation of 48,000 answers from 10 LLM configurations.
• Development of SeqSnap, a custom GPT-4o configuration using NCBI APIs.

Main Results:

• SeqSnap achieved the best overall performance among evaluated LLMs.
• GPT-4o with web access and GeneGPT showed complementary strengths.
• LLMs demonstrate both promise and limitations in current genomic applications.

Conclusions:

• GeneTuring provides a valuable resource for benchmarking and advancing LLMs in genomics.
• Integrating LLMs with domain-specific tools like SeqSnap enhances genomic intelligence.
• Further research is needed to overcome current LLM limitations in complex genomic tasks.