Related Experiment Videos
SparseVLM+: Visual Token Sparsification With Improved Text-Visual Attention Pattern
Summary
This study introduces SparseVLM, a novel training-free method to optimize visual tokens in vision-language models (VLMs). SparseVLM enhances efficiency by reducing computational overhead without sacrificing accuracy, making VLMs more accessible.
Area of Science:
- Computer Science
- Artificial Intelligence
- Machine Learning
Background:
- Vision-language models (VLMs) face computational challenges due to high overhead from visual tokens.
- Existing methods for visual token reduction often require extensive training data and fine-tuning.
- Information sparsity in visual tokens compared to text tokens necessitates efficient processing strategies.
Purpose of the Study:
- To develop a training-free mechanism for optimizing visual tokens in VLMs, reducing computational costs without performance degradation.
- To address the limitations of existing token pruning methods by eliminating the need for fine-tuning.
- To enhance the efficiency and effectiveness of VLMs through intelligent token selection and compression.
Main Methods:
- Proposed SparseVLM, a text-guided, training-free token optimization mechanism for VLMs.
- Utilized self-attention matrices to rate visual token significance based on relevant text tokens.
- Introduced a rank-based strategy for adaptive sparsification ratios and a token recycling method for compressing pruned tokens.
- Mitigated Attention Gravity and Attention Sink phenomena with improved text-visual attention patterns.
- Developed Temporal-Balanced Sparsification for video understanding tasks to reduce cross-frame redundancy.
Main Results:
- SparseVLM+ framework demonstrated effectiveness across 23 image and video understanding tasks and 7 VLMs.
- Equipping LLaVA with SparseVLM+ resulted in a 57.1% FLOPs reduction and 38.4% CUDA latency decrease.
- Maintained an average of 99.6% of the original accuracy after optimization.
- Significantly reduced computational overhead while preserving model performance.
Conclusions:
- SparseVLM offers an efficient, training-free solution for optimizing visual tokens in VLMs.
- The proposed methods effectively address challenges like Attention Gravity and Attention Sink, improving text-visual alignment.
- SparseVLM+ framework shows broad applicability and significant performance gains in both image and video understanding tasks.
