Pre-training Genomic Language Model with Variants for Better Modeling Functional Genomics.

Area of Science:

• Genomics
• Computational Biology
• Bioinformatics

Background:

• Genomic language models (GLMs) learn from DNA sequences to represent genomic context.
• Sequence-to-function (S2F) models link genetic information to gene expression and phenotypes.
• Bridging GLMs and S2F models for individualized gene expression prediction remains a challenge.

Purpose of the Study:

• To develop a novel DNA language model, UKBioBERT, using UK BioBank genetic data.
• To integrate UKBioBERT with existing S2F models (Enformer, Borzoi) to create enhanced predictive models.
• To improve the prediction of gene expression levels and understand the impact of genetic variants.

Main Methods:

• Pre-training a DNA language model (UKBioBERT) on genetic variants from the UK BioBank.
• Generating informative sequence embeddings from UKBioBERT.
• Combining UKBioBERT embeddings with S2F architectures (Enformer, Borzoi) to form UKBioFormer and UKBioZoi.
• Evaluating model performance on gene expression prediction across different cohorts.

Main Results:

• UKBioBERT embeddings effectively identify gene functions and improve gene expression prediction in cell lines.
• UKBioFormer and UKBioZoi demonstrate superior performance in predicting highly predictable gene expression levels.
• The integrated models generalize well across diverse cohorts.
• UKBioFormer accurately captures genotype-phenotype relationships, enabling in-silico mutation analysis.

Conclusions:

• Integrating genomic language models with sequence-to-function approaches significantly advances functional genomics.
• UKBioBERT provides valuable embeddings for understanding gene function and expression predictability.
• The developed UKBioFormer and UKBioZoi models offer improved tools for predicting gene expression and analyzing genetic variant effects.