BiRNA-BERT allows efficient RNA language modeling with adaptive tokenization.

Area of Science:

• Computational Biology
• Bioinformatics
• Genomics

Background:

• Transformer models show promise in biological sequence analysis but struggle with RNA's long sequences.
• Current RNA language models often truncate sequences, losing critical distal nucleotide context.
• Standard NLP tokenization methods are ill-suited for nucleotide-level RNA analysis.

Purpose of the Study:

• To develop a Transformer model, BiRNA-BERT, capable of processing arbitrarily long RNA sequences.
• To introduce an adaptive dual-tokenization framework for RNA sequence modeling.
• To achieve state-of-the-art performance in RNA-related tasks, including structural prediction.

Main Methods:

• Developed BiRNA-BERT, a 117M-parameter Transformer encoder trained on 36 million non-coding RNA sequences.
• Implemented an adaptive dual-tokenization framework combining nucleotide-level (NUC) and byte-pair encoding (BPE).
• Dynamically selected tokenization strategy based on input sequence length to avoid truncation.

Main Results:

• BiRNA-BERT processed arbitrarily long RNA sequences without truncation, outperforming existing models.
• Achieved state-of-the-art results in tasks from short-sequence classification to long-context modeling and RNA structural prediction.
• Demonstrated strong intrinsic language modeling performance (perplexity, token recovery) with a compact model size.

Conclusions:

• BiRNA-BERT effectively addresses RNA sequence length constraints in Transformer models.
• The adaptive dual-tokenization framework enhances RNA sequence modeling capabilities.
• BiRNA-BERT offers a powerful and efficient solution for diverse RNA analysis tasks.