Short-Context Regulatory DNA Language Models with Motif-Discovery Regularization

Area of Science:

• Genomics
• Computational Biology
• Bioinformatics

Background:

• Self-supervised DNA language models (DNALMs) trained on whole genomes struggle with sparse, heterogeneous regulatory sequences and short motifs.
• Existing annotation-agnostic DNALMs often underperform simpler models on regulatory tasks due to difficulties in learning regulatory syntax.

Purpose of the Study:

• Introduce ARSENAL, a short-context masked DNA language model optimized for regulatory sequence analysis.
• Improve the discovery of transcription factor motifs and prediction of regulatory variant effects.
• Enhance supervised learning tasks like chromatin accessibility prediction and regulatory variant scoring.

Main Methods:

• Trained ARSENAL on a functionally enriched regulatory corpus with a novel regularizer promoting motif discovery.
• Evaluated ARSENAL's performance in zero-shot motif recovery and regulatory variant effect prediction.
• Integrated ARSENAL embeddings into supervised models for chromatin accessibility prediction and variant scoring.

Main Results:

• ARSENAL demonstrated superior recovery of transcription factor motifs *de novo* and improved prediction of regulatory variant effects compared to other DNALMs.
• Incorporating ARSENAL embeddings significantly boosted supervised chromatin accessibility prediction accuracy across multiple cell types.
• ARSENAL embeddings led to enhanced regulatory variant scoring and enabled targeted regulatory sequence design.

Conclusions:

• ARSENAL effectively addresses limitations of large-scale DNALMs in capturing regulatory sequence features.
• The model provides a powerful tool for motif discovery, variant effect prediction, and functional genomics analysis.
• ARSENAL facilitates the design of regulatory sequences with specific functional properties.