用瓦瑟斯坦远程学习进行自我监督学习的实证研究.
Makoto Yamada1,2, Yuki Takezawa1,3, Guillaume Houry1,4
1Machine Learning and Data Science Unit, Okinawa Institute of Science and Technology, Okinawa 904-0412, Japan.
Entropy (Basel, Switzerland)
|November 27, 2024
概括
本研究探讨了使用树-瓦瑟斯坦距离 (TWD) 的自我监督学习 (SSL). 我们发现,将TWD与特定的概率模型和杰弗里分歧规范化相结合,可以稳定训练,并提高表现.
科学领域:
- 机器学习 机器学习
- 计算机视觉 计算机视觉
背景情况:
- 自主监督学习 (SSL) 通常使用等号相似性.
- 瓦瑟斯坦距离,特别是树-瓦瑟斯坦距离 (TWD),在SSL中较少被探索.
- 训练瓦斯斯坦距离可以在数量上具有挑战性.
研究的目的:
- 研究使用TWD优化SSL.
- 在SSL中确定TWD的稳定培训程序.
- 评估TWD在表示学习中对等号相似性的有效性.
主要方法:
- 雇佣了两种TWD类型:总变化和集群树.
- 测试了各种概率模型:软max,ArcFace和简化嵌入.
- 介绍了杰弗里基于分歧的规范化,以优化稳定性.
主要成果:
- 软max和TWD组合在基准数据集 (STL10,CIFAR10/100,SVHN) 上表现优于标准的SimCLR.
- 使用SimSiam的TWD未能训练,表明对模型组合的敏感性.
- 杰弗里分歧规范化显著帮助了模型训练.
结论:
- 在SSL与TWD中的模型性能高度依赖于所选择的概率模型.
- 当TWD与概率模型和正规化相适当结合时,它超越了基于共弦相似性的表示学习方法.
相关概念视频
Wald-Wolfowitz Runs Test I
609
The Wald-Wolfowitz test, also known as the runs test, is a nonparametric statistical test used to assess the randomness of a sequence of two different types of elements (e.g., positive/negative values, successes/failures). It examines whether the order of the elements in a sequence is random or if there is a pattern or trend present. This nonparametric test applies to any ordered data despite the population and sample data distribution, even if a higher sample size is available.
The test works...
The test works...
609
One-Compartment Open Model: Wagner-Nelson and Loo Riegelman Method for ka Estimation
393
This lesson introduces two critical methods in pharmacokinetics, the Wagner-Nelson and Loo-Riegelman methods, used for estimating the absorption rate constant (ka) for drugs administered via non-intravenous routes. The Wagner-Nelson method relates ka to the plasma concentration derived from the slope of a semilog percent unabsorbed time plot. However, it is limited to drugs with one-compartment kinetics and can be impacted by factors like gastrointestinal motility or enzymatic degradation.
On...
On...
393
Empirical Method to Interpret Standard Deviation
5.1K
The empirical rule, also known as the three-sigma rule, allows a statistician to interpret the standard deviation in a normally distributed dataset. The rule states that 68% of the data lies within one standard deviation from the mean, 95% lies within two standard deviations from the mean, and 99.7% lies within three standard deviations from the mean. Additionally, this rule is also called the 68-95-99.7 rule.
This rule is used widely in statistics to calculate the proportion of data values...
This rule is used widely in statistics to calculate the proportion of data values...
5.1K
Mean Absolute Deviation
2.6K
The mean absolute deviation is also a measure of the variability of data in a sample. It is the absolute value of the average difference between the data values and the mean.
Let us consider a dataset containing the number of unsold cupcakes in five shops: 10, 15, 8, 7, and 10. Initially, calculate the sample mean. Then calculate the deviation, or the difference, between each data value and the mean. Next, the absolute values of these deviations are added and divided by the sample size to...
Let us consider a dataset containing the number of unsold cupcakes in five shops: 10, 15, 8, 7, and 10. Initially, calculate the sample mean. Then calculate the deviation, or the difference, between each data value and the mean. Next, the absolute values of these deviations are added and divided by the sample size to...
2.6K
Kendall's Coefficient of Concordance
257
Kendall's Coefficient of Concordance (W), also known as Kendall's W, is a non-parametric statistical measure used to assess the agreement or concordance between multiple raters or judges when they rank a set of items. It is often used when you have ordinal data (ranks) and you want to see if there is consistency or consensus among the raters. It is widely applied in research areas such as psychology, medicine, and social sciences, where multiple judges are asked to rank or rate subjects...
257
Wilcoxon Signed-Ranks Test for Matched Pairs
87
The Wilcoxon signed-rank test for matched pairs evaluates the null hypothesis by combining the ranks of differences with their signs. It essentially tests whether the median of the differences in a population of matched pairs is zero. Since the test incorporates more information than the sign test, it generally yields more trustable conclusions. This test also does not require the data to follow a normal distribution, but two conditions must be met for it to be applicable: (1) the data must...
87


