Hessian-guided mixed-precision quantization of vision transformers under hardware constraints

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViTs) offer strong global modeling but suffer from high computational complexity and parameter count, hindering deployment on resource-constrained devices.
• Model quantization is a key technique for reducing ViT inference overhead, yet existing methods struggle to balance accuracy with hardware efficiency.
• Efficient deployment of ViTs on edge devices remains a significant challenge due to their resource demands.

Purpose of the Study:

• To propose a novel mixed-precision quantization framework specifically designed for Vision Transformers (ViTs) under hardware constraints.
• To improve the adaptability and efficiency of ViT model compression techniques.
• To enable the effective deployment of ViTs on hardware with limited resources.

Main Methods:

• Implemented a mixed-precision quantization framework utilizing both uniform and non-uniform quantization strategies based on activation value distributions.
• Introduced a Hessian-based sensitivity metric to guide the allocation of quantization bit widths by assessing layer sensitivity to quantization errors.
• Formulated the mixed-precision bit-width allocation as an optimal binary mask search problem, solved using integer linear programming to meet hardware constraints.

Main Results:

• The proposed framework effectively balances model accuracy, size, and inference latency.
• Achieved significant reductions in model size and computational cost compared to existing methods on benchmark datasets like ImageNet and COCO.
• Demonstrated competitive accuracy while enhancing hardware deployment efficiency for ViTs.

Conclusions:

• The developed mixed-precision quantization framework offers a viable solution for deploying Vision Transformers on resource-constrained hardware.
• The Hessian-based sensitivity metric and integer linear programming approach provide an effective strategy for optimizing bit-width allocation.
• This work contributes to making powerful ViT models more accessible for practical applications in edge computing and embedded systems.