A foundation language model to decipher diverse regulation of RNAs

RNA metabolism is tightly regulated by cis-elements and trans-acting factors. Most information guiding such regulation is encoded in RNA sequences. Deciphering the regulatory rules is critical for RNA biology and therapeutics; however, the prediction of diverse regulation from RNA sequences remains a formidable challenge.

Considering the similarities in semantic and syntactic features between RNAs and human language, we present LAMAR, a transformer-based foundation LAnguage Model for RNA Regulation, to decipher general rules underlying RNA processing. The model is pretrained on approximately 15 million sequences from both genome and transcriptome of 225 mammals and 1569 viruses, and further fine-tuned with labeled datasets for various tasks. The resulting fine-tuned models outperform the state-of-the-art methods in predicting mRNA translation efficiency and mRNA half-life, while achieving comparable accuracy to specifically designed methods in predicting splice sites of pre-mRNAs and internal ribosome entry sites (IRESs). The fine-tuned LAMAR is further applied to predict mutational effects of cis-regulatory elements and reveals known and novel regulatory elements that modulate RNA degradation. The fine-tuned LAMAR is also applied in an in silico screen of novel IRESs, resulting in the identifications of highly active IRESs that promote circRNA translation.

Our results indicate that a single foundation language model is applicable in the comprehensive analysis of different aspects of RNA regulation and predictive identification of novel regulatory elements, providing new insight into the design and optimization of RNA drugs.