DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA.

Area of Science:

• Genomics
• Computational Biology
• Bioinformatics

Background:

• Self-supervised models have advanced NLP, vision, and protein analysis.
• Genomic DNA language models (DNALMs) aim to represent DNA elements for genomic tasks.
• Existing benchmarks lack focus on non-coding regulatory DNA elements.

Purpose of the Study:

• Introduce DART-Eval, a benchmark suite for evaluating DNALMs on regulatory DNA.
• Assess DNALM performance in zero-shot, probed, and fine-tuned settings.
• Compare DNALMs against contemporary *ab initio* models.

Main Methods:

• Developed DART-Eval, focusing on biologically relevant tasks.
• Included sequence motif discovery, cell-type specific regulatory activity prediction, and counterfactual prediction.
• Evaluated annotation-agnostic DNALMs against baseline models.

Main Results:

• Current DNALMs show inconsistent performance on regulatory DNA tasks.
• DNALMs did not consistently outperform baseline models.
• DNALMs required significantly more computational resources than baselines.

Conclusions:

• Annotation-agnostic DNALMs require further development for regulatory genomics.
• Future DNALMs may benefit from improved modeling, data curation, and evaluation strategies.
• DART-Eval provides a framework for advancing DNALM research.