Benchmarking DNA foundation models for genomic and genetic tasks.

Area of Science:

• Genomics
• Bioinformatics
• Machine Learning

Background:

• DNA foundation models are rapidly advancing genomics.
• Comprehensive evaluations of these models are currently lacking.
• Understanding model performance across diverse tasks is essential.

Purpose of the Study:

• To provide a comprehensive, unbiased benchmark of five leading DNA foundation models.
• To evaluate model performance on diverse genomic and genetic tasks using zero-shot embeddings.
• To identify optimal embedding strategies for different genomic applications.

Main Methods:

• Benchmarked five DNA foundation models: DNABERT-2, Nucleotide Transformer V2, HyenaDNA, Caduceus-Ph, and GROVER.
• Evaluated models on sequence classification, gene expression prediction, variant effect quantification, and TAD region recognition.
• Utilized zero-shot embeddings and compared various pooling strategies, focusing on mean token embedding.

Main Results:

• Mean token embedding significantly improved sequence classification performance compared to other strategies.
• General-purpose DNA foundation models performed well in pathogenic variant identification.
• Models showed limitations in predicting gene expression and identifying causal QTLs compared to specialized approaches.

Conclusions:

• Model performance is task- and dataset-dependent.
• Mean token embedding is a robust strategy for sequence classification.
• Findings provide a framework for selecting DNA foundation models based on task requirements, architecture, pre-training data, and embedding strategies.