A Dual-Language-Model Framework for Reproducibility in Small Molecule-RNA Binding Site Prediction

Area of Science:

• Computational biology
• Machine learning in drug discovery
• Bioinformatics

Background:

• Single-seed evaluation is common in small-dataset molecular learning but can inflate performance estimates.
• Reproducibility in RNA-ligand binding site prediction using large pretrained RNA language models is underexplored.

Purpose of the Study:

• To conduct the first systematic reproducibility analysis for RNA-ligand binding site prediction.
• To integrate two large pretrained RNA language models (RNA-FM and RiNALMo) across multiple fusion architectures.
• To assess performance over replicated training runs on the TR60/TE18 benchmark.

Main Methods:

• Utilized two large pretrained RNA language models: RNA-FM and RiNALMo.
• Implemented multiple fusion architectures, including Reverse Cross-Attention and simple concat fusion.
• Performed replicated training runs on the TR60/TE18 benchmark dataset.
• Analyzed performance using Matthews Correlation Coefficient (MCC) and mean accuracy with standard deviation.

Main Results:

• A Peak-SOTA Paradox was observed, where a single-seed run (MCC 0.353) surpassed reported state-of-the-art, while multi-seed replication yielded a lower average (0.266 ± 0.020), a 32.8% overestimation.
• Mean accuracy was consistent across architectures, but reproducibility varied significantly.
• Simple concat fusion strategies showed higher stability than attention-based models under data scarcity.
• Single-seed evaluations can overstate expected performance by 20-30% in limited-sample regimes.

Conclusions:

• Reproducibility should be a primary evaluation criterion for small-sample molecular prediction.
• A dual-reporting standard is motivated: mean ± SD as the principal metric and peak scores as supplementary evidence.
• Architectural choices, not just parameter count, influence variance in low-data scenarios.
• Variance-aware evaluation is essential to avoid misrepresenting model performance in limited-sample settings.