OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Federated learning (FL) enables collaborative model training on edge devices, preserving data privacy.
• Vision Transformers (ViTs) show promise but are often too large and computationally intensive for resource-constrained FL environments.
• Existing lightweight models struggle to balance efficiency, adaptability to non-IID data, and performance in FL.

Purpose of the Study:

• To develop efficient and lightweight vision models suitable for on-device federated learning.
• To address the challenges of limited resources, data heterogeneity, and communication bottlenecks in FL.
• To propose OnDev-LCT, a novel architecture for on-device vision tasks.

Main Methods:

• Proposed OnDev-LCT models featuring a Lightweight Convolutional Transformer (LCT) tokenizer.
• Incorporated image-specific inductive biases using depthwise separable convolutions in residual linear bottleneck blocks for local feature extraction.
• Utilized multi-head self-attention (MHSA) in the LCT encoder for global representation learning.

Main Results:

• OnDev-LCT models demonstrated superior performance compared to existing lightweight vision models on benchmark datasets.
• Achieved lower parameter counts and reduced computational demands.
• Showed suitability for federated learning scenarios with data heterogeneity and communication constraints.

Conclusions:

• OnDev-LCT offers an effective solution for deploying vision models on resource-limited edge devices within federated learning frameworks.
• The proposed architecture successfully balances model efficiency, computational performance, and adaptability to diverse data distributions.
• Enables wider adoption of advanced vision models in privacy-preserving, distributed machine learning applications.