Integrating convolution and self-attention improves language model of human genome for interpreting non-coding...

Area of Science:

• Genomics
• Computational Biology
• Bioinformatics

Background:

• Interpreting the non-coding genome is crucial for human genetics but challenging due to the vastness of biochemical elements.
• Deep learning offers promising computational approaches for non-coding region interpretation.

Purpose of the Study:

• To develop a computationally efficient deep learning model, LOGO (Language of Genome), for interpreting the human non-coding genome.
• To enhance sequence labeling and variant prioritization tasks using self-supervised learning on the human reference genome.

Main Methods:

• Developed LOGO, a self-attention based contextualized pre-trained language model with a light architecture (1 million parameters).
• Employed self-supervision for bidirectional genome representation learning.
• Fine-tuned LOGO for sequence labeling and extended it for variant prioritization using convolutional modules and special input encoding.

Main Results:

• Achieved 15% absolute improvement in promoter identification and 4.5% in enhancer-promoter interaction prediction.
• Demonstrated state-of-the-art multi-task predictive power on thousands of chromatin features with significantly fewer parameters than existing models.
• Improved sensitivity and specificity in prioritizing non-coding variants associated with human diseases.

Conclusions:

• LOGO provides an accurate, fast, scalable, and robust framework for interpreting non-coding genomic regions.
• The model successfully inferred regulatory mechanisms for type 2 diabetes GWAS signals.
• LOGO advances base-resolution interpretation of the non-coding genome for sequence labeling and variant prioritization.