RiNALMo: general-purpose RNA language models can generalize well on structure prediction tasks.

Area of Science:

• Bioinformatics
• Computational Biology
• Molecular Biology

Background:

• Ribonucleic acid (RNA) is emerging as a key target for small-molecule drugs, necessitating a deeper understanding of its structure and function.
• Vast amounts of unlabeled RNA sequence data generated by sequencing technologies hold significant untapped potential for biological insights.
• Existing deep learning methods struggle with generalizing RNA secondary structure predictions to novel RNA families.

Purpose of the Study:

• To introduce the RiboNucleic Acid Language Model (RiNALMo), the largest RNA language model developed to date.
• To leverage advances in protein language models for analyzing RNA sequences.
• To extract implicit structural information and hidden knowledge from large RNA datasets.

Main Methods:

• Pre-training RiNALMo, a 650M parameter model, on a dataset of 36 million non-coding RNA sequences from diverse databases.
• Utilizing a transformer-based architecture, similar to successful protein language models.
• Evaluating RiNALMo's performance on various downstream tasks, including secondary structure prediction.

Main Results:

• RiNALMo achieved state-of-the-art performance across multiple RNA-related downstream tasks.
• Demonstrated superior generalization capabilities compared to existing deep learning models, particularly for predicting secondary structures of unseen RNA families.
• Successfully captured implicit structural information embedded within RNA sequences.

Conclusions:

• RiNALMo represents a significant advancement in analyzing RNA sequences and understanding their functions.
• The model's ability to generalize to new RNA families addresses a critical limitation in current deep learning approaches for RNA structure prediction.
• RiNALMo unlocks the potential of large unlabeled RNA datasets for drug discovery and biological research.