MMViT: Bridging Mamba and Attention for efficient video action recognition in sports

In sports AI, human action recognition (HAR) faces a challenge between the expensive Transformer and the one-dimensional state space models (SSMs). Although Transformer has proven success on video tasks, its high computational cost scales quadratically. In contrast, conventional SSMs like Mamba possess linear complexity, but also underperform in the recognition. In this paper, we propose MMViT (Multi-scale Mamba Visual Transformer) with a hierarchical design for improved recognition and objective efficiency. We employ a heterogeneous "Attention-Mamba-Attention" (A-M-A) strategy. It first uses Multi-scale Pooling Attention (MPA) for efficient capture of local spatial feature. As computation-heavy stages come, it transitions to Mamba module with linear complexity to efficiently model long-range temporal context. Finally, attention is re-introduced at latter stages for semantic feature fusion. Also, we introduce a computation-downsampling decoupling (CDD) mechanism to preserve feature coverage during Mamba spatial scaling change. We have validated our approach on SpaceJam and Basketball-51 datasets. Experiments show that MMViT achieves superior performance over strong baselines with substantial margins. Ablation studies show the significance of A-M-A, MPA and CDD. MMViT achieves competitive accuracy among evaluated models and provides a favorable accuracy-efficiency trade-off for video action recognition task.