在Dev-LCT:在设备上的轻量级卷积变压器朝着联合学习的方向发展.
Chu Myaet Thwal1, Minh N H Nguyen2, Ye Lin Tun1
1Department of Computer Science and Engineering, Kyung Hee University, Yongin-si, Gyeonggi-do 17104, South Korea.
概括
我们介绍了OnDev-LCT,轻量级的卷积变压器,旨在在联合学习 (FL) 中实现高效的设备视觉任务. 这些模型平衡了尺寸和性能,优于资源有限的边缘设备的现有轻量级选项.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 联合学习 (FL) 允许在边缘设备上进行协作模型训练,从而保护数据隐私.
- 视觉变压器 (ViT) 是有前途的,但对于资源有限的FL环境来说,它们往往太大,计算密集.
- 现有的轻量级模型难以平衡效率,适应非IID数据的适应性和在FL中的性能.
研究的目的:
- 开发适合在设备上联合学习的高效和轻量级视觉模型.
- 为了应对有限资源,数据异质性和FL的通信瓶的挑战.
- 提出OnDev-LCT,一个用于设备视觉任务的新架构.
主要方法:
- 拟议的OnDev-LCT模型具有轻量级卷积变压器 (LCT) 代币化器.
- 整合了特定图像的诱导偏差,使用在残余线性瓶块中的深度可分离的卷积来进行局部特征提取.
- 在LCT编码器中利用多头自我注意力 (MHSA) 进行全球表示学习.
主要成果:
- 与基准数据集上现有的轻量级视觉模型相比,OnDev-LCT模型表现出更高的性能.
- 实现了更低的参数计数和更低的计算需求.
- 显示适用于数据异质性和通信约束的联合学习场景.
结论:
- 在联合学习框架内,OnDev-LCT为在资源有限的边缘设备上部署视觉模型提供了有效的解决方案.
- 拟议的架构成功地平衡了模型效率,计算性能和适应各种数据分布的适应性.
- 允许在保护隐私的分布式机器学习应用中更广泛地采用先进的视觉模型.
相关概念视频
01:22Force Classification
1.2K
Forces play a crucial role in the study of physics and engineering. They are essential in describing the motion, behavior, and equilibrium of objects in the physical world. Forces can be classified based on their origin, type, and direction of action.
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
Contact and non-contact forces are two of the most widely used categories of forces. As the name suggests, contact forces require physical contact between two objects to act upon each other. Examples of contact forces include frictional,...
1.2K
01:16Types Of Transformers
979
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
979
01:18Introduction to Learning
438
Learning is the process of acquiring knowledge or skills through practice or experience, leading to long-lasting behavioral changes. This acquisition occurs through interaction with the environment and requires practice or experience. For instance, mastering a skill such as surfing requires considerable practice and experience, highlighting the essential role of repeated interactions with the environment in learning.
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
In contrast to learned behaviors, unlearned behaviors such as crying, sexual...
438
01:26Transformers
1.1K
A device that transforms voltages from one value to another using induction is called a transformer. A transformer consists of two separate coils, or windings, wrapped around the same soft iron core. However, they are electrically insulated from each other.
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
The iron core has a substantial relative permeability. Therefore, the magnetic field lines generated due to the current in one winding are almost entirely confined within the core, such that the same magnetic flux permeates each turn of both...
1.1K
01:27Transformers in Distribution System
103
Transformers in distribution systems can be broadly categorized into distribution substation transformers and other distribution transformers. They are crucial for stepping down high transmission voltages to levels suitable for distribution and end-user applications.
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
Distribution substation transformers come in various ratings and typically use mineral oil for insulation and cooling. To prevent moisture and air from entering the oil, some transformers use an inert gas like nitrogen to fill the...
103
01:21Cognitive Learning
243
Cognitive learning is based on purposive behavior, incidental learning, and insight learning.
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
E. C. Tolman's theory of purposive behavior emphasizes that much behavior is goal-directed. He argued that to understand behavior, we must look at the entire sequence of actions leading to a goal. For instance, high school students study hard, not just due to past reinforcement but also to achieve the goal of getting into a good college.
Tolman introduced the idea that behavior is influenced by...
243