Evaluation of large language models for discovery of gene set function

Area of Science:

• Genomics
• Bioinformatics
• Artificial Intelligence

Background:

• Functional genomics relies on gene function databases, which are often incomplete.
• Discovering gene functions from omics data is crucial for biological insights.

Purpose of the Study:

• To evaluate the ability of large language models (LLMs) to identify common gene functions from gene sets.
• To assess LLMs' capacity for providing molecular rationale and self-confidence scores for identified functions.

Main Methods:

• Five LLMs (GPT-4, GPT-3.5, Gemini Pro, Mixtral Instruct, Llama2 70b) were tested on curated Gene Ontology gene sets and gene clusters from omics data.
• Performance was measured by function similarity to curated names, confidence assessment accuracy for random gene sets, specificity, and gene coverage.

Main Results:

• GPT-4 accurately suggested functions for 73% of curated gene sets and showed high confidence for correct (random) gene sets (87%).
• Other LLMs exhibited variable function recovery and were often falsely confident for random sets.
• For omics data clusters, GPT-4 identified functions in 45% of cases with high specificity and gene coverage, supported by verifiable rationale and citations.

Conclusions:

• LLMs, particularly GPT-4, show potential as valuable assistants in functional genomics and omics data analysis.
• GPT-4's self-confidence assessment is a reliable indicator of function prediction accuracy.
• LLMs offer a complementary approach to traditional methods, enhancing specificity and gene coverage in function discovery.