A multi-modal transformer for cell type-agnostic regulatory predictions.

Area of Science:

• Genomics
• Computational Biology
• Epigenetics

Background:

• Deep learning models excel at analyzing genomic sequences for cis-regulatory elements.
• Current models struggle to generalize predictions to different cellular environments.
• Understanding gene regulation requires integrating sequence information with cell type-specific data.

Purpose of the Study:

• To develop a deep learning model that generalizes across cellular contexts for regulatory genomics.
• To create a multi-modal transformer that combines genomic sequence and chromatin accessibility data.
• To improve the interpretability and predictive power of models in regulatory genomics.

Main Methods:

• Developed EpiBERT, a multi-modal transformer model.
• Employed a masked accessibility-based pre-training objective.
• Fine-tuned EpiBERT for gene expression prediction and other regulatory genomics tasks.

Main Results:

• EpiBERT achieves gene expression prediction accuracy comparable to sequence-only models like Enformer.
• EpiBERT demonstrates significant generalization to unobserved cell states.
• Learned representations are interpretable and aid in predicting caQTLs, motifs, and enhancer-gene links.

Conclusions:

• EpiBERT represents a significant advancement in creating generalizable deep learning models for regulatory genomics.
• Integrating multi-modal data improves model performance and applicability across diverse cellular contexts.
• This work paves the way for more robust sequence-based deep learning in genomics.