卷积器与变压器在蛋白质序列预训练方面具有竞争力.
Kevin K Yang1, Nicolo Fusi1, Alex X Lu1
1Microsoft Research New England, Cambridge, MA 02139, USA.
Cell systems
|March 1, 2024
概括
卷积神经网络 (CNN) 为蛋白质语言建模提供了变压器模型的有效替代方案. 在各种任务中,CNN甚至在较长的蛋白质序列中实现了竞争性性能,提高了计算效率.
科学领域:
- 生物信息学是一种生物信息学.
- 计算生物学 计算生物学
- 机器学习 机器学习
背景情况:
- 预训练的蛋白质序列模型增强生物信息学任务,但经常使用变压器架构.
- 变压器模型具有二进制缩放限制,限制蛋白质序列长度.
- 当前最先进的模型面临着序列长度的约束.
研究的目的:
- 调查卷积神经网络 (CNN) 架构是否可以匹配蛋白质语言模型中的变压器效率.
- 探索CNN的线性缩放,以处理更长的蛋白质序列.
- 评估CNN在蛋白质序列建模方面的表现.
主要方法:
- 利用CNN架构的掩面语言模型预训.
- 将CNN的性能与下游预测任务中的变压器模型进行比较.
- 对其处理超出电流变压器极限的序列的能力进行了评估.
主要成果:
- 与变压器相比,CNN表现出具有竞争力的,有时甚至更优异的性能.
- 在长于变压器模型所能处理的蛋白质序列上,CNN保持了强的性能.
- 这项研究证实了CNN的线性缩放与序列长度.
结论:
- CNN架构是用于蛋白质语言建模的变压器的可行和高效的替代方案.
- 蛋白质建模中的计算效率可以通过使用CNN来提高,而不会造成性能损失.
- 从模型架构中解脱预训练任务对于推进蛋白质语言模型至关重要.
相关概念视频
Conservation of Protein Domains Over Different Proteins
10.9K
Protein domains are small structurally independent units that are part of a single amino acid chain. Although these domains are often structurally independent, they may rely on synergistic effects to perform their functions as part of a larger protein. Protein domains may be conserved within the same organism, as well as across different organisms.
A limited set of protein domains often duplicate and recombine during evolution. These domains can be organized in different combinations to...
A limited set of protein domains often duplicate and recombine during evolution. These domains can be organized in different combinations to...
10.9K
Transformers with Off-Nominal Turns Ratios
152
In scenarios involving parallel transformers with disparate ratings, developing per-unit models requires accommodating off-nominal turns ratios. This situation arises when the selected base voltages are not proportional to the transformer’s voltage ratings. Consider a transformer where the rated voltages are related by the term a. If the chosen voltage bases satisfy a relationship involving term b, term c is defined as the ratio of these bases. This ratio is then substituted into the...
152
Types Of Transformers
975
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
975
Convolution Properties I
149
Convolution computations can be simplified by utilizing their inherent properties.
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
The commutative property reveals that the input and the impulse response of an LTI (Linear Time-Invariant) system can be interchanged without affecting the output:
149
Convolution: Math, Graphics, and Discrete Signals
254
In any LTI (Linear Time-Invariant) system, the convolution of two signals is denoted using a convolution operator, assuming all initial conditions are zero. The convolution integral can be divided into two parts: the zero-input or natural response and the zero-state or forced response, with t0 indicating the initial time.
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
To simplify the convolution integral, it is assumed that both the input signal and impulse response are zero for negative time values. The graphical convolution process...
254
Improving Translational Accuracy
10.4K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
10.4K


