Evaluation of large language models for discovery of gene set function

Area of Science:

• Functional genomics
• Bioinformatics
• Computational biology

Background:

• Current gene function databases are incomplete, limiting functional genomics.
• Gene set analysis requires accurate identification of common biological functions.
• Evaluating the utility of Large Language Models (LLMs) for gene set analysis is crucial.

Approach:

• Assessed five LLMs (GPT-4, Gemini-Pro, Mixtral-Instruct, Llama2-70b) for gene function discovery.
• Benchmarked LLM performance against curated Gene Ontology gene sets and random gene sets.
• Evaluated LLM ability to provide rationale, citations, and confidence scores for identified functions.
• Tested LLMs on gene sets derived from 'omics data to identify novel functions.

Key Points:

• GPT-4 demonstrated high confidence in identifying functions for canonical gene sets (73%) and zero confidence for random sets.
• Gemini-Pro and Mixtral-Instruct showed naming ability but were overconfident on random sets; Llama2-70b performed poorly.
• GPT-4 identified novel, verifiable gene functions from 'omics data not found by traditional methods (32% of cases).

Conclusions:

• LLMs, particularly GPT-4, show significant promise as tools for functional genomics and 'omics data analysis.
• LLMs can rapidly synthesize common gene functions, acting as valuable assistants for researchers.
• Further development and validation are needed, but LLMs offer a powerful new approach to supplement existing functional enrichment methods.