Detecting and quantifying overparametrization in RNA language models with REDIAL

Area of Science:

• Computational Biology
• Bioinformatics
• Machine Learning for RNA

Background:

• RNA language models (LMs) are crucial foundation models (FMs) for RNA structural prediction.
• Current evaluation methods using supervised tasks can obscure FM inefficiencies and memorization.
• A need exists for unsupervised frameworks to assess RNA LM performance directly.

Purpose of the Study:

• Introduce REDIAL (RNA Embedding perturbation Diagnostics for Language models), a zero-shot, unsupervised framework.
• Evaluate how RNA LMs internalize structural constraints by analyzing their latent spaces.
• Identify architectural design flaws and parameterization inefficiencies in current RNA LMs.

Main Methods:

• Developed REDIAL, an unsupervised framework for extracting coevolutionary signals from RNA LM latent spaces.
• Performed layer-wise dissection and ablation studies on popular RNA LMs.
• Compared structure-guided pretraining with sequence-only baselines.

Main Results:

• Uncovered significant layer-wise disparities in how RNA LMs learn structural constraints.
• Demonstrated that RNA LMs are often overparameterized relative to available RNA sequence diversity, leading to inefficiency and overfitting.
• Showed that structure-guided pretraining enhances the signal-to-noise ratio of coevolutionary couplings compared to sequence-only methods.

Conclusions:

• REDIAL exposes critical flaws in current RNA LM parameter scaling strategies.
• Current RNA LM architectures exhibit design flaws leading to parameter inefficiency.
• This unsupervised benchmark guides the development of more efficient and generalizable RNA FMs for therapeutics and de novo design.