在低级张量空间中学习全球提示符,用于异质联合学习
Lele Fu1, Sheng Huang1, Yuecheng Li2
1School of Systems Science and Engineering, Sun Yat-sen University, Guangzhou, China.
概括
联邦学习挑战由FedGPT解决,它使用低级张量提示来减少通信和处理数据/模型异质性. 这种方法显著提高了全球模型的性能和效率.
科学领域:
- 人工智能的人工智能
- 机器学习 机器学习
- 分布式系统 分布式系统
背景情况:
- 联合学习 (FL) 允许在没有数据传输的情况下进行协作模式培训,确保隐私和安全.
- 目前的FL方法在通信开销,非IID数据和模型异质性方面扎.
- 这些挑战限制了FL在现实应用中的可扩展性和有效性.
研究的目的:
- 提出FedGPT,一个新的联合学习框架,利用低级张量空间中的全球提示.
- 在联合学习中解决通信负担,数据异质性和模型异质性.
- 提高全球模型的概括性和在异质环境中的性能.
主要方法:
- 使用提示符而不是模型参数来转移知识,以减少通信量.
- 使用第三阶张量和张量奇数值分解 (SVD) 来从异质客户端提示中提取全球信息.
- 通过提示来提高性能,使不同尺寸的本地模型之间的知识转移成为可能.
主要成果:
- 美联储GPT显著优于最先进的方法,达到高达13.21%的改进.
- 通信量减少到不到FedAvg.所要求的3%以下.
- 该框架有效地处理联合学习中的数据和模型异质性.
结论:
- 通过利用低等级张量提示,FedGPT为异质联合学习提供了卓越的解决方案.
- 拟议的方法大大降低了通信开销,同时提高了模型性能.
- FedGPT展示了高效和有效的协作机器学习的巨大潜力.
相关概念视频
Improving Translational Accuracy
8.5K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
8.5K
Forced Transdifferentiation
1.9K
Transdifferentiation, also known as lineage reprogramming, was first discovered by Selman and Kafatos in 1974 in silkmoths. They observed that the moths’ cuticle-producing cells transformed into salt-producing cells. Many such cases of natural transdifferentiation occur in organisms. In humans, pancreatic alpha cells can become beta cells. In newts, the loss of the eye’s lens causes the pigmented epithelial cells to transdifferentiate into the lens cells.
Artificial...
Artificial...
1.9K
Multi-input and Multi-variable systems
93
Cruise control systems in cars are designed as multi-input systems to maintain a driver's desired speed while compensating for external disturbances such as changes in terrain. The block diagram for a cruise control system typically includes two main inputs: the desired speed set by the driver and any external disturbances, such as the incline of the road. By adjusting the engine throttle, the system maintains the vehicle's speed as close to the desired value as possible.
In the absence...
In the absence...
93
Associative Learning
275
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
275
Friedman Two-way Analysis of Variance by Ranks
130
Friedman's Two-Way Analysis of Variance by Ranks is a nonparametric test designed to identify differences across multiple test attempts when traditional assumptions of normality and equal variances do not apply. Unlike conventional ANOVA, which requires normally distributed data with equal variances, Friedman's test is ideal for ordinal or non-normally distributed data, making it particularly useful for analyzing dependent samples, such as matched subjects over time or repeated measures...
130
Parallel Processing
143
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
143


