Comprehensive benchmarking of large language models for RNA secondary structure prediction

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning

Background:

• Large language models (LLMs) have shown success in modeling DNA and protein sequences.
• LLMs for RNA are emerging, learning rich numerical representations of RNA bases from massive datasets.
• High-quality RNA representations are hypothesized to improve performance on data-intensive tasks like RNA secondary structure prediction.

Purpose of the Study:

• To comprehensively analyze and compare pretrained RNA LLMs for RNA secondary structure prediction.
• To evaluate the generalization capabilities of RNA LLMs on new and diverse RNA structures.
• To establish a unified experimental setup and curated benchmark datasets for RNA LLM evaluation.

Main Methods:

• Utilized a common deep learning architecture to assess RNA LLM representations for secondary structure prediction.
• Evaluated multiple pretrained RNA LLMs on benchmark datasets with increasing generalization difficulty.
• Conducted a comparative analysis of RNA LLM performance across different homology levels.

Main Results:

• Two specific RNA LLMs demonstrated superior performance compared to other models.
• Significant challenges were identified in the generalization capabilities of RNA LLMs, particularly in low-homology scenarios.
• The study provides curated benchmark datasets and a unified experimental framework.

Conclusions:

• Pretrained RNA LLMs can enhance RNA secondary structure prediction, but their generalization abilities require further investigation.
• Model selection is critical, as performance varies significantly among different RNA LLMs.
• Future research should focus on improving RNA LLM generalization, especially for sequences with low homology to training data.