A comprehensive evaluation of large language models in mining gene relations and pathway knowledge

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Manual literature curation struggles to keep pace with biological discoveries.
• Large language models (LLMs) trained on vast text corpora hold potential for extracting biological knowledge.
• LLMs can be utilized as biological knowledge graphs for analyzing gene networks and pathways.

Purpose of the Study:

• To evaluate the performance of 21 large language models (LLMs) in retrieving biological knowledge.
• To assess LLMs' capabilities in predicting gene regulatory relations (activation, inhibition, phosphorylation) and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway components.
• To compare the effectiveness of API-based versus open-source LLMs for biological knowledge extraction.

Main Methods:

• Assessed 21 LLMs, including API-based and open-source models.
• Focused evaluation on predicting gene regulatory relations and identifying KEGG pathway components.
• Quantified performance using F1 scores for gene regulatory relations and Jaccard similarity for KEGG pathways.

Main Results:

• Significant performance disparities were observed among the evaluated LLMs.
• API-based models GPT-4 and Claude-Pro demonstrated superior performance in both tasks.
• Open-source models generally lagged behind, with Falcon-180b and llama2-7b showing the highest scores among them.

Conclusions:

• LLMs are valuable tools for gene network analysis and pathway mapping, offering insights into biological knowledge extraction.
• Model selection is critical, as performance varies significantly between different LLMs.
• This study provides a benchmark and case study for leveraging LLMs as knowledge graphs in biological research.