ProtRNA: A protein-derived RNA language model by cross-modality transfer learning.

Area of Science:

• Bioinformatics
• Computational Biology
• Genomics

Background:

• Protein language models (PLMs) like ESM-2 excel at analyzing protein sequences.
• RNA language models face challenges due to limited and less conserved RNA sequences.
• Effective transfer of knowledge between biological sequence modalities is an open research question.

Purpose of the Study:

• To investigate if information from PLMs can be effectively transferred to improve RNA sequence analysis.
• To develop a model that addresses the challenges of low-resource RNA data.
• To evaluate the performance of cross-modality transfer learning in biological language models.

Main Methods:

• Developed ProtRNA, a model employing a cross-modality transfer learning strategy.
• Adapted the ESM-2 protein language model to process RNA sequences.
• Leveraged evolutionary and physicochemical information from protein sequences for RNA data.

Main Results:

• ProtRNA demonstrated comparable or superior performance on various RNA downstream tasks.
• The model achieved these results using significantly less trainable parameters (1/8) and training data (1/6) compared to a baseline RNA language model.
• Successfully adapted a high-performing PLM for a "low-resource" biological sequence modality.

Conclusions:

• Cross-modality transfer learning is a viable and effective strategy for enhancing RNA language models.
• ProtRNA showcases the potential of leveraging knowledge from well-resourced domains (proteins) to address challenges in under-resourced domains (RNA).
• This approach offers a promising direction for developing more efficient and powerful biological language models.