Benchmarking large language models for genomic knowledge with GeneTuring.

Area of Science:

• Genomics
• Artificial Intelligence
• Biomedical Research

Background:

• Large language models (LLMs) show potential in biomedical research.
• The utility of LLMs as a knowledge base for genomic research is largely unexplored.

Purpose of the Study:

• To evaluate the performance of leading LLMs in answering genomic research questions.
• To assess the reliability of LLMs for genomic data inquiry.

Main Methods:

• Development of GeneTuring, a Q&A database with 1,200 genomics questions.
• Manual scoring of 25,200 answers generated by six LLMs (including GPT-4o, Claude 3.5, Gemini Advanced).

Main Results:

• GPT-4o, with web access, demonstrated the highest overall performance.
• GPT-4o excelled in most genomic question-answering tasks compared to other models.
• Despite strong performance, GPT-4o did not answer all questions correctly.

Conclusions:

• LLMs, including advanced models like GPT-4o, are not yet fully reliable for genomic inquiries.
• Further development is needed to ensure accuracy and completeness of LLM-generated genomic information.
• LLMs may serve as a supplementary tool but require careful validation in genomic research.