Retentive Network promotes efficient RNA language modeling of long sequences

Area of Science:

• Computational Biology
• Bioinformatics
• Genomics

Background:

• Latent features of RNA sequences are vital for understanding their functions.
• Transformer models are widely used for nucleotide language modeling but have O(n²) complexity, limiting long sequence processing.

Purpose of the Study:

• To propose RNAret, an efficient RNA language model with O(n) complexity.
• To enable effective processing of long RNA sequences for feature extraction and biological insights.

Main Methods:

• Developed RNAret based on the Retention Network mechanism.
• Pretrained RNAret using self-supervised masked language modeling on 29.8 million RNA sequences.
• Achieved training parallelism, low computational overhead, and long-sequence processing.

Main Results:

• RNAret demonstrated superior performance on RNA-RNA interaction prediction.
• Achieved high accuracy in RNA secondary structure prediction.
• Showcased effectiveness in mRNA/lncRNA classification tasks.

Conclusions:

• RNAret is a powerful RNA language model with O(n) complexity, suitable for long sequences.
• The model effectively extracts latent features from RNA sequences.
• RNAret holds significant potential for advancing RNA biology research.