Benchmarking large language models for identifying transcription factor regulatory interactions | JoVE Visualize

Area of Science:

Computational Biology
Genomics
Bioinformatics

Background:

Transcription factors (TFs) and their target genes orchestrate gene expression, influencing biological processes and diseases.
Existing methods for identifying TF-target interactions often require specialized computational expertise.
Large language models (LLMs) offer a more accessible approach to querying these complex regulatory relationships.

Purpose of the Study:

To benchmark the performance of prominent LLMs in identifying human TF-target interactions.
To evaluate the impact of prompt engineering and model parameters on LLM accuracy for TF-target predictions.
To assess LLM capabilities across different regulatory interaction types (bidirectional, ambiguous, self-regulated, unidirectional).

Main Methods:

Benchmarking four LLMs (Claude 3.5 Sonnet, Gemini 1.0 Pro, GPT-4o, Llama3 8b) using literature-curated (8432 interactions) and experimentally derived (5148 interactions) human TF-target datasets.
Analyzing performance based on single-turn vs. multi-turn queries and zero-temperature settings.
Evaluating accuracy across four regulatory categories: bidirectional, ambiguous, self-regulated, and unidirectional.

Main Results:

Claude 3.5 Sonnet and GPT-4o showed competitive performance in single-turn queries.
Multi-turn prompting significantly improved accuracy for some models, notably Claude 3.5 Sonnet on self-regulated pairs (+32.6%).
Excluding unknown regulation types improved accuracy, with unidirectional regulation reaching nearly 70% balanced accuracy.
Claude 3.5 Sonnet consistently outperformed other models across various conditions with experimentally derived interactions.

Conclusions:

LLM chatbots, particularly Claude 3.5 Sonnet, show significant potential for TF-target interaction identification in regulatory biology.
Prompt engineering and parameter tuning are crucial for optimizing LLM performance in this domain.
These findings establish a benchmarking framework and highlight the utility of general-purpose LLMs for researchers lacking specialized computational expertise.