特性规范化防止了非对比性学习动态的崩.
1Institute of Statistical Mathematics, Kyoto 6068501, Japan bao.han@ism.ac.jp.
Neural computation
|August 14, 2025
概括
特性正常化对于防止非对比学习的崩至关重要. 这项研究表明,与L2损失不同,共数损失诱导了稳定的动态,确保了没有负面示例的强有力的表示学习.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 计算机科学 计算机科学
背景情况:
- 相反的学习使用正面和负面的例子来学习表示.
- 像BYOL和SimSiam这样的非对比式学习方法消除了效率的负面例子.
- 以前的理论表明,强大的数据增强可以防止非对比性学习的崩.
研究的目的:
- 用特征规范化分析非对比学习的动态.
- 扩展现有理论,通过结合等位数损失和特征正常化来扩展现有理论.
- 了解特征规范化如何防止表示崩.
主要方法:
- 学习动态的理论分析.
- 以L2损失为基础的理论扩展到等号损失.
- 对表示崩和稳定的平衡进行数学建模.
主要成果:
- 与L2损失不同的是,代数损失会诱导高阶 (第六阶) 动态.
- 特性规范化被证明是防止表示崩的关键.
- 一个稳定的平衡出现动态,即使最初崩的解决方案.
结论:
- 特征正常化在防止非对比性学习崩方面发挥着至关重要的作用.
- 输入符号损失,包括特征正常化,提供了一个更稳定的学习动态.
- 这项工作为自我监督表示学习提供了新的理论见解.
相关概念视频
Generalization, Discrimination, and Extinction
788
Generalization, discrimination, and extinction are key concepts in operant conditioning that influence how behaviors are learned and maintained.
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
Generalization occurs when a behavior reinforced in one context is performed in similar situations. For instance, a student who studies diligently for calculus and receives excellent grades might apply the same study habits to psychology and history, expecting similar results. Generalization shows how learning in one setting can influence behavior in...
788
Associative Learning
575
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
575
Survival Tree
159
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
159
Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model
126
Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
126
Introduction to Learning
532
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
532
Normal and Tangetial Components: Problem Solving
224
Consider a man with a mass of 70 kg seated in a chair connected to a pin support through a member BC. If the man maintains an upright position, the task is to determine the horizontal and vertical reactions of the chair on the man when the member makes a 45° angle with the horizontal. At this moment, the man has a speed of 5 m/s, increasing at a rate of 1 m/s².
224


