基于透的字节补丁变压器用于自主监督的SMILES数据预训练
Medard Edmund Mswahili1, JunHa Hwang1, Kyuri Jo1
1Chungbuk National University, Department of Computer Engineering, Cheongju 28644, South Korea.
iScience
|February 2, 2026
概括
新型的SMILES字节补丁变压器 (SMiBPT) 通过自适应地细分化学字符串来增强分子表示,改善化学学习中的大语言模型性能.
科学领域:
- 计算化学的计算化学
- 机器学习 机器学习
- 生物信息学是一种生物信息学.
背景情况:
- 变压器模型正在推进分子表示学习.
- 捕捉局部化和层次化化学结构仍然是当前模型面临的挑战.
研究的目的:
- 引入SMILES字节补丁变压器 (SMiBPT) 以改进分子表示学习.
- 开发一种适应性模型,用于化学链的动态细分.
主要方法:
- SMiBPT使用基于的字节补丁来将SMILES和DeepSMILES字符串分割成具有化学意义的子结构.
- 该模型集成了自我监督的预训练,化学动机感知编码,适应性感知掩盖和旋转位置嵌入.
- 在PubChem的21600万个未标记的分子上训练,没有截断.
主要成果:
- 在预测准确性和效率方面,SMiBPT的表现优于ChemBERTa,SMILES-BERT和MoLFormer等现有模型.
- 适应性补丁策略保留了分子语义,并增强了特征提取.
- 展示了卓越的零射击传输能力.
结论:
- SMiBPT提供了一个参数高效和有效的方法来学习分子表示.
- 适应性细分方法解决了化学学习中固定代币化的局限性.
- 这种模型推进了化学中大型语言模型的应用.
相关概念视频
Entropy
36.1K
Salt particles that have dissolved in water never spontaneously come back together in solution to reform solid particles. Moreover, a gas that has expanded in a vacuum remains dispersed and never spontaneously reassembles. The unidirectional nature of these phenomena is the result of a thermodynamic state function called entropy (S). Entropy is the measure of the extent to which the energy is dispersed throughout a system, or in other words, it is proportional to the degree of disorder of a...
36.1K
Entropy
3.6K
The first law of thermodynamics is quantitatively formulated via an equation relating the internal energy of a system, the heat exchanged by it, and the work done on it. A quantitative formulation of the second law of thermodynamics leads to defining a state function, the entropy.
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
When an ideal gas expands isothermally, the disorder in the gas increases. From the molecular perspective, the gas molecules have more volume to move around in.
Consider an infinitesimal step in the expansion, which...
3.6K
Long-patch Base Excision Repair
8.0K
Since the discovery of the two BER pathways, there has been a debate about how a cell chooses one pathway over the other and the factors determining this selection. Numerous in vitro experiments have pointed out multiple determinants for the sub-pathway selection. These are:
8.0K
Standard Entropy Change for a Reaction
24.6K
Entropy is a state function, so the standard entropy change for a chemical reaction (ΔS°rxn) can be calculated from the difference in standard entropy between the products and the reactants.
24.6K
Entropy and Solvation
8.4K
The process of surrounding a solute with solvent is called solvation. It involves evenly distributing the solute within the solvent. The rule of thumb for determining a solvent for a given compound is that like dissolves like. A good solvent has molecular characteristics similar to those of the compound to be dissolved. For example, polar solutions dissolve polar solutes, and apolar solvents dissolve apolar solutes. A polar solvent is a solvent that has a high dielectric constant (ϵ...
8.4K
Entropy within the Cell
12.9K
A living cell's primary tasks of obtaining, transforming, and using energy to do work may seem simple. However, the second law of thermodynamics explains why these tasks are harder than they appear. None of the energy transfers in the universe are completely efficient. In every energy transfer, some amount of energy is lost in a form that is unusable. In most cases, this form is heat energy. Thermodynamically, heat energy is defined as the energy transferred from one system to another that...
12.9K


