对个性化联合学习的标志度规范化
1Department of Computing and Information Technology, The University of the West Indies, St. Augustine 350462, Trinidad and Tobago.
Entropy (Basel, Switzerland)
|June 26, 2025
概括
标志-规范化 (SER) 通过稳定客户端-本地优化来增强个性化联合学习. 这种新的方法减少了梯度符号的变化,从而提高了分布式系统的准确性和更快的融合.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 分布式系统 分布式系统
背景情况:
- 个性化联合学习 (PFL) 解决了在异质分布式数据上培训客户特定模型的挑战.
- 由于数据异质性,现有的PFL方法经常在优化稳定性和个性化有效性方面扎.
研究的目的:
- 引入标志度规范化 (SER),这是一种新的技术,用于提高个性化和稳定性在联合学习.
- 通过惩罚局部优化中的过度方向变化来增强客户特定的模型培训.
主要方法:
- 开发了一种基于的新型规范化技术,符号-规范化 (SER),灵感来自笛卡尔的符号规则.
- 在梯度符号分布上定义了一个可微分的信号率目标,并将其集成到FedAvg和FedProx中.
- 在不修改通信协议的情况下,在每次本地回合中有效地应用了SER.
主要成果:
- 在FEMNIST,莎士比亚和CIFAR-10数据集中,SER显著提高了平均和最坏情况下的客户端准确性.
- 证明了客户之间的差异减少,加速了趋同,并平滑了当地的损失表面.
- 在比较评估中表现优于最先进的个性化方法,如Ditto和pFedMe.
结论:
- 符号-规则化提供了一个可扩展和正交的机制,用于增强联邦学习中的个性化.
- 通过信息理论和几何规范化,SER稳定了学习动态,适用于资源受限的设置.
- 该方法显示了基于轨迹的规范化和混合引导优化在联合学习中的潜力.
相关概念视频
Entropy and the Second Law of Thermodynamics
3.2K
The second law of thermodynamics can be stated quantitatively using the concept of entropy. Entropy is the measure of disorder of the system.
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
The relation between entropy and disorder can be illustrated with the example of the phase change of ice to water. In ice, the molecules are located at specific sites giving a solid state, whereas, in a liquid form, these molecules are much freer to move. The molecular arrangement has therefore become more randomized. Although the change in average...
3.2K
Propagation of Uncertainty from Random Error
1.1K
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
1.1K
Survival Tree
166
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
166
Entropy Change in Reversible Processes
2.7K
In the Carnot engine, which achieves the maximum efficiency between two reservoirs of fixed temperatures, the total change in entropy is zero. The observation can be generalized by considering any reversible cyclic process consisting of many Carnot cycles. Thus, it can be stated that the total entropy change of any ideal reversible cycle is zero.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
The statement can be further generalized to prove that entropy is a state function. Take a cyclic process between any two points on a p-V diagram.
2.7K
Randomized Experiments
7.9K
The randomization process involves assigning study participants randomly to experimental or control groups based on their probability of being equally assigned. Randomization is meant to eliminate selection bias and balance known and unknown confounding factors so that the control group is similar to the treatment group as much as possible. A computer program and a random number generator can be used to assign participants to groups in a way that minimizes bias.
Simple randomization
Simple...
Simple randomization
Simple...
7.9K
Standard Entropy Change for a Reaction
21.4K
Entropy is a state function, so the standard entropy change for a chemical reaction (ΔS°rxn) can be calculated from the difference in standard entropy between the products and the reactants.
21.4K

