Taylor-Series-Expansion-Based Vision Transformer Models.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Taylor-Series Expansion (TSE) approximates nonlinear functions using finite series.
• Vision transformers are powerful deep learning models for image analysis.
• Naive vision transformers can be computationally intensive and memory-demanding.

Purpose of the Study:

• To design a novel TSE-based vision transformer that reduces memory footprint and computational cost.
• To maintain or improve accuracy compared to standard vision transformers.
• To enhance deployment performance and latency.

Main Methods:

• Developed a TSE-based vision transformer block inspired by the first-order Taylor series term.
• Incorporated learnable TSE coefficients and finite multiplications to approximate naive transformer operations.
• Introduced a Taylor skip mechanism during training for dynamic expansion capabilities.

Main Results:

• TSE-based models achieved significant boosts in deployment latency (1.30-1.45×) on A100 and AGX Orin GPUs with negligible accuracy loss.
• Performance gains were observed across ImageNet classification, COCO detection, and ADE20K segmentation tasks.
• Combined with model compression, TSE optimization further enhanced latency and throughput by up to 1.87× and 3.61×, respectively.

Conclusions:

• The TSE-based vision transformer offers an efficient alternative to naive models, reducing memory burden without sacrificing accuracy.
• The model demonstrates strong dynamic expansion capabilities and significant real-world deployment advantages.
• TSE-based optimization is orthogonal to model compression, enabling synergistic performance improvements.