Distillation-free Scaling of Large State-Space Models for Images and Videos

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• State-space models (SSMs) offer novel context modeling in deep learning.
• Mamba improves context modeling with data-dependent matrices but faces parameter scalability challenges in vision applications.

Purpose of the Study:

• To address the parameter scalability limitations of Mamba-based architectures for image classification and action recognition.
• To enhance the robustness and performance of large-scale SSMs without additional techniques like knowledge distillation.

Main Methods:

• Analysis of Mamba-based and Attention-based model characteristics.
• Proposal of a novel Mamba-Attention interleaved architecture.
• Evaluation on ImageNet-1K, Kinetics-400, and Something-Something-v2 benchmarks.

Main Results:

• The proposed interleaved architecture enhances scalability and robustness of Mamba-based models.
• Improved performance on image classification and action recognition tasks.
• Up to +1.7% accuracy improvement over state-of-the-art Mamba-based architectures.

Conclusions:

• The Mamba-Attention interleaved architecture effectively resolves scalability issues in Mamba-based models for vision tasks.
• The architecture demonstrates increased robustness to common corruption artifacts.
• This approach offers a promising direction for advancing large-scale SSMs in computer vision.