ERNIE-RNA: an RNA language model with structure-enhanced representations

Area of Science:

• Computational Biology
• Bioinformatics
• Genomics

Background:

• Existing RNA language models (RLMs) often neglect crucial structural information in RNA sequences.
• This oversight leads to incomplete feature extraction and suboptimal performance on various biological tasks.

Purpose of the Study:

• To introduce ERNIE-RNA, an enhanced RNA language model incorporating base-pairing restrictions.
• To demonstrate ERNIE-RNA's superior capability in capturing RNA structural features and improving downstream task performance.

Main Methods:

• ERNIE-RNA is built upon a modified Bidirectional Encoder Representations from Transformers (BERT) architecture.
• The model utilizes attention mechanisms to capture RNA structural information, enabling zero-shot prediction.

Main Results:

• ERNIE-RNA's attention maps show enhanced ability to capture RNA structural features compared to traditional methods like RNAfold and RNAstructure.
• Fine-tuned ERNIE-RNA achieves state-of-the-art results in RNA structure and function prediction tasks.
• The model demonstrates natural development of comprehensive RNA architecture representations during pre-training.

Conclusions:

• ERNIE-RNA offers versatile features for diverse RNA research applications.
• Integrating knowledge-based priors into BERT-like frameworks can significantly improve language model performance for biological sequences.