探测基因组语言模型:核酸生成预训练变压器和预训练在学习表征中的作用
Shae M Mclaughlin1, Daniel A Lim2,3,4
1Department of Epidemiology and Biostatistics, University of California, San Francisco, 550 16th St., Floor 2, San Francisco, CA 94158, United States.
Briefings in bioinformatics
|February 3, 2026
概括
基因组语言模型 (gLMs) 从预训练中获益,但重复的元素可能会扭曲学习. 优化用于重复序列的预训练策略对于捕捉更广泛的基因组语法和生物功能至关重要.
科学领域:
- 基因组学就是基因组学.
- 计算生物学 计算生物学
- 机器学习 机器学习
背景情况:
- 基因组语言模型 (gLMs) 在预训练期间学习的表示的可解释性正在积极研究中.
- 重复元素 (RE) 构成了哺乳动物基因组的很大一部分,可能会影响模型训练.
研究的目的:
- 调查预训练对gLMs的影响以及基因组数据中重复元素的作用.
- 为基因组序列分析开发和评估一种新型仅用解码器的变压器模型,即核酸生成预训练变压器 (GPT).
主要方法:
- 使用了一种只有解码器的变压器,并使用单核酸代码化.
- 在预训练期间,对重复元素的重量进行了实验变化.
- 进行了比较微调,对内部表示进行线性探测,以及基于稀疏自编码器 (SAE) 的可解释性分析.
主要成果:
- 具有REs (0.5) 中度下加权的模型在七个基因组分类任务中表现出最佳性能.
- 预训练模型的表现明显优于随机初始化的基线.
- SAE分析揭示了模型表示中的RE关联模式,表明它们在预训练目标中占主导地位.
结论:
- 预训对于gLMs来说是有价值的,但目前的策略可能会过度强调重复的序列.
- 未来的预训练方法应该平衡常见的重复序列的学习与不太频繁的,具有生物学意义的模式.
- 解决学习功能性基因组语法超越无处不在的重复的挑战对于推动gLMs至关重要.
相关概念视频
Genomics
40.7K
Genomics is the science of genomes: it is the study of all the genetic material of an organism. In humans, the genome consists of information carried in 23 pairs of chromosomes in the nucleus, as well as mitochondrial DNA. In genomics, both coding and non-coding DNA is sequenced and analyzed. Genomics allows a better understanding of all living things, their evolution, and their diversity. It has a myriad of uses: for example, to build phylogenetic trees, to improve productivity and...
40.7K
Nucleotide Excision Repair
40.9K
Overview
40.9K
Language
917
Language is a unique communication system that uses words and systematic rules to organize and transmit information. Unlike other forms of communication, which may involve postures, movements, odors, or vocalizations, language relies on symbols and grammar. This makes human communication distinct from that of other species, who also communicate but do not use language in the same way humans do.
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
Corballis and Suddendorf (2007) and Tomasello and Rakoczy (2003) highlight the role of language in...
917
State Space Representation
581
The frequency-domain technique, commonly used in analyzing and designing feedback control systems, is effective for linear, time-invariant systems. However, it falls short when dealing with nonlinear, time-varying, and multiple-input multiple-output systems. The time-domain or state-space approach addresses these limitations by utilizing state variables to construct simultaneous, first-order differential equations, known as state equations, for an nth-order system.
Consider an RLC circuit, a...
Consider an RLC circuit, a...
581
Graphical Representation of Inequalities
218
The graph of the equation where y equals x squared forms a curve known as a parabola. This curve acts as a boundary in the coordinate plane, dividing it into distinct regions based on the relative position of points.When the equality sign in the equation is replaced with an inequality—such as greater than, less than, greater than or equal to, or less than or equal to—the graphical representation changes from a single curve into a broader shaded area that signifies the set of all...
218
Control Volume and System Representations
1.6K
Two key frameworks are employed to analyze mass, energy, and momentum transfer: the control volume approach and the system approach. These frameworks offer different perspectives, depending on whether the focus is on a specific region in space (control volume approach) or a defined mass of fluid (system approach).
The control volume approach considers a stationary region in space through which fluid flows. This region is bounded by a control surface. For instance, in the case of water...
The control volume approach considers a stationary region in space through which fluid flows. This region is bounded by a control surface. For instance, in the case of water...
1.6K


