A Comprehensive Evaluation of Large Language Models in Mining Gene Interactions and Pathway Knowledge

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Automated text mining of biological pathways is crucial for understanding disease mechanisms and drug development.
• Manual literature curation struggles to keep pace with the rapid growth of scientific publications.
• Large language models (LLMs) offer potential for efficient pathway analysis due to their extensive training data.

Approach:

• Evaluated 21 LLMs (API-based and open-source) for biological pathway analysis.
• Focused on predicting gene regulatory relations (activation, inhibition, phosphorylation) and recognizing KEGG pathway components.
• Utilized precision, recall, F1 scores, and Jaccard similarity for performance assessment.

Key Points:

• API-based models, particularly ChatGPT-4 and Claude-Pro, demonstrated superior performance in gene regulatory relation prediction and KEGG pathway recognition.
• Open-source models, including Falcon-180b-chat and llama1-7b, showed lower but notable performance.
• Significant performance disparities exist among evaluated LLMs.

Conclusions:

• LLMs are valuable tools for gene network analysis and pathway mapping in biomedical research.
• Careful selection of LLMs is essential due to varying effectiveness.
• This study provides insights into using LLMs as knowledge graphs for biological pathway data.