Benchmarking of signaling networks generated by large language models

Area of Science:

• Computational biology
• Systems biology
• Bioinformatics

Background:

• Computational models of molecular signaling networks are crucial for understanding cell decisions.
• Manual curation of these models from scientific literature is time-consuming and often incomplete.

Purpose of the Study:

• To evaluate the capability of general-purpose large language models (LLMs) in generating accurate computational models of cellular signaling networks.
• To establish benchmarks for LLM performance in this domain.

Main Methods:

• Testing general-purpose LLMs to generate signaling network models for specific biological processes (cardiomyocyte hypertrophy, myofibroblast activation, mechano-signaling).
• Comparing LLM-generated network components against manually curated, literature-derived networks.
• Assessing the accuracy of LLM-generated networks in predicting cellular responses to perturbations.

Main Results:

• General-purpose LLMs generated 24-58% of reactions found in literature-curated signaling networks.
• The accuracy of predicting network responses to perturbations ranged from 5-26%.

Conclusions:

• Current general-purpose LLMs show limited accuracy in generating comprehensive and precise signaling network models.
• This research provides a foundational pipeline and performance benchmarks for advancing LLM-based approaches in computational systems biology.