Benchmarking DNA large language models on quadruplexes

Area of Science:

• Genomics
• Bioinformatics
• Computational Biology

Background:

• Large language models (LLMs) show promise in predicting genomic elements.
• Selecting the optimal LLM for specific tasks like whole-genome annotation remains challenging.
• LLMs in genomics are broadly categorized into transformer-based, long convolution-based, and state-space models (SSMs).

Purpose of the Study:

• To benchmark different large language model (LLM) architectures for whole-genome annotation of G-quadruplexes (GQ).
• To evaluate the performance of transformer-based, long convolution-based, and state-space models in identifying GQ structures.
• To determine which LLM architectures are best suited for specific downstream genomic tasks.

Main Methods:

• Benchmarking three LLM architectures (transformer-based, long convolution-based, SSMs) for whole-genome G-quadruplex (GQ) mapping.
• Evaluating model performance using F1 and Matthews Correlation Coefficient (MCC) metrics.
• Analyzing whole-genome annotations to identify distinct functional elements recovered by each model type.

Main Results:

• All evaluated LLMs performed comparably, with DNABERT-2 and HyenaDNA showing superior F1 and MCC scores.
• HyenaDNA demonstrated enhanced recovery of quadruplexes in distal enhancers and intronic regions.
• Different LLM architectures, particularly HyenaDNA and Caduceus versus transformer-based models, showed distinct patterns in de novo quadruplex generation.

Conclusions:

• LLM architectures with varying context lengths can detect distinct functional regulatory elements.
• The choice of LLM architecture is crucial for specific genomic tasks, as different models offer complementary strengths.
• This study highlights the importance of selecting appropriate LLMs for accurate and comprehensive whole-genome annotation.