Toward Unified Expertise: Learning a Single Vision Model From Diverse Perception.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Multi-task learning (MTL) faces optimization challenges from conflicting gradients.
• Parameter sharing in MTL can hinder task-specific representation learning.
• Existing large models often prioritize efficiency over adjustable efficiency.

Purpose of the Study:

• To introduce Mod-Squad, a modular transformer model for MTL that balances cooperation and specialization.
• To extend Mod-Squad for multi-dataset pre-training across heterogeneous sources.
• To develop efficient adaptation techniques for flexible finetuning of modular models.

Main Methods:

• Proposed Mod-Squad, a modular transformer with a sparse subset of experts activated per task.
• Introduced a novel mutual information-based loss for differentiable matching and unifying heterogeneous data.
• Developed efficient adaptation techniques for dynamic adjustment of model size, parameters, and computational cost.

Main Results:

• Mod-Squad effectively balances task cooperation and specialization, avoiding full backbone sharing.
• The model demonstrates scalability with increasing tasks and dataset sizes.
• Achieved favorable performance-efficiency trade-offs through hybrid adaptation schemes.

Conclusions:

• Mod-Squad provides a robust foundation for sparse modular models capable of learning from diverse data and supervision.
• The emergent modularity facilitates strong generalization, component decomposition, and efficient adaptation.
• The proposed methods enable dynamic adjustment of model efficiency for downstream tasks.