MADTP++: Bridge the Gap Between Token and Weight Pruning for Accelerating VLTs.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Vision-Language Transformers (VLTs) show great potential but suffer from high computational costs.
• Current compression methods for VLTs are limited, often ignoring cross-modal alignment and dynamic compression needs.

Purpose of the Study:

• To develop a novel, unified framework for efficient compression of Vision-Language Transformers.
• To address the limitations of existing methods in handling token and weight pruning simultaneously.

Main Methods:

• Proposed MADTP++, integrating Multi-modality Alignment Guidance (MAG) and Dynamic Token Pruning (DTP) for token compression.
• Introduced Hardware-aware Weight Pruning (HWP) utilizing Sparse Tensor Cores for fine-grained weight pruning.
• Implemented a Cooperative Optimization Training Strategy with Knowledge Distillation Constraints for joint optimization.

Main Results:

• MADTP++ significantly reduces model parameters and computational costs (GFLOPs).
• The method achieves superior compression compared to existing VLT compression techniques.
• Experiments show competitive performance is maintained across various VLT models and datasets.

Conclusions:

• MADTP++ provides an effective and unified approach for compressing Vision-Language Transformers.
• The framework enables significant efficiency gains without compromising model performance.
• The proposed method offers a flexible and hardware-aware solution for VLT model optimization.