BiRNA-BERT允许高效的RNA语言建模与自适应的令牌化
Md Toki Tahmid1, Haz Sameen Shahgir2, Sazan Mahbub3
1Department of Computer Science and Engineering, Bangladesh University of Engineering and Technology, Dhaka, Bangladesh.
Communications biology
|November 20, 2025
概括
BiRNA-BERT使用自适应式双标记化克服了RNA序列长度限制,使得长RNA序列的分析能够在没有数据丢失的情况下进行. 这个变压器模型在RNA结构预测和语言建模方面实现了最先进的性能.
科学领域:
- 计算生物学 计算生物学
- 生物信息学是一种生物信息学.
- 基因组学就是基因组学.
背景情况:
- 变压器模型在生物序列分析方面表现有前途,但在RNA的长序列方面存在困难.
- 当前的RNA语言模型通常会切断序列,失去关键的远端核酸语境.
- 标准的NLP标记化方法不适合核酸水平RNA分析.
研究的目的:
- 开发一个变压器模型,BiRNA-BERT,能够处理任意长的RNA序列.
- 为RNA序列建模引入一个自适应的双标记化框架.
- 在RNA相关任务中实现最先进的性能,包括结构预测.
主要方法:
- 开发了BiRNA-BERT,一个117M参数的变压器编码器,经过3600万个非编码RNA序列的训练.
- 实施了一个自适应的双标记化框架,结合了核酸水平 (NUC) 和字节对编码 (BPE).
- 基于输入序列长度的动态选择的代币化策略,以避免截断.
主要成果:
- BiRNA-BERT在没有截断的情况下处理任意长的RNA序列,优于现有的模型.
- 在从短序分类到长文本建模和RNA结构预测等任务中取得了最先进的结果.
- 展示了强大的内在语言建模性能 (困惑性,令牌恢复) 与一个紧的模型大小.
结论:
- 双RNA-BERT有效地解决了变压器模型中的RNA序列长度约束.
- 适应式双标记化框架增强了RNA序列建模能力.
- BiRNA-BERT为各种RNA分析任务提供了强大而高效的解决方案.
相关概念视频
01:02Long-patch Base Excision Repair
7.8K
Since the discovery of the two BER pathways, there has been a debate about how a cell chooses one pathway over the other and the factors determining this selection. Numerous in vitro experiments have pointed out multiple determinants for the sub-pathway selection. These are:
7.8K
02:23RNA Editing
9.7K
RNA editing is a post-transcriptional modification where a precursor mRNA (pre-mRNA) nucleotide sequence is changed by base insertion, deletion, or modification. The extent of RNA editing varies from a few hundred bases, in mitochondrial DNA of trypanosomes, to a just single base, in nuclear genes of mammals. Even a single base change in the pre-mRNA can convert a codon for one amino acid into the codon for another amino acid or a stop codon. This type of re-coding can significantly affect the...
9.7K
01:23RNA Interference
27.7K
RNA interference (RNAi) is a process in which a small non-coding RNA molecule blocks the post-transcriptional expression of a gene by binding to its messenger RNA (mRNA) and preventing the protein from being translated.
This process occurs naturally in cells, often through the activity of genomically-encoded microRNAs. Researchers can take advantage of this mechanism by introducing synthetic RNAs to deactivate specific genes for research or therapeutic purposes. For example, RNAi could be used...
This process occurs naturally in cells, often through the activity of genomically-encoded microRNAs. Researchers can take advantage of this mechanism by introducing synthetic RNAs to deactivate specific genes for research or therapeutic purposes. For example, RNAi could be used...
27.7K
03:21RNA-seq
11.7K
RNA sequencing, or RNA-Seq, is a high-throughput sequencing technology used to study the transcriptome of a cell. Transcriptomics helps to interpret the functional elements of a genome and identify the molecular constituents of an organism. Additionally, it also helps in understanding the development of an organism and the occurrence of diseases.
Before the discovery of RNA-seq, microarray-based methods and Sanger sequencing were used for transcriptome analysis. However, while...
Before the discovery of RNA-seq, microarray-based methods and Sanger sequencing were used for transcriptome analysis. However, while...
11.7K
00:43Bacterial RNA Polymerase
32.4K
Unlike eukaryotes, bacteria use a single RNA Polymerase (RNAP) to transcribe all genes. The different subunits of bacterial RNAPhave distinct functions. The multisubunit structure of the bacterial RNAP helps the enzyme to maintain catalytic function, facilitate assembly, interact with DNA and RNA, and self-regulate its activity.
In most genes, the transcription site is a single base present upstream of the coding sequence. Though RNAP is a catalytically efficient enzyme, it does not recognize...
In most genes, the transcription site is a single base present upstream of the coding sequence. Though RNAP is a catalytically efficient enzyme, it does not recognize...
32.4K
00:43Bacterial RNA Polymerase
11.5K
11.5K