S $^{2}$ Q-VDiT$^+$: Accurate Quantized Video Diffusion Transformer with Multi-Resolution Sampling and Structural...

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Large-scale video diffusion models (V-DMs) generate high-quality videos but suffer from significant computational costs, hindering deployment.
• Existing Post-Training Quantization (PTQ) methods are often ineffective for modern V-DMs due to their multi-resolution nature, leading to calibration issues.
• Current PTQ pipelines typically calibrate at a single resolution, failing to address the inherent multi-resolution characteristics of V-DMs trained in stages.

Purpose of the Study:

• To develop an efficient Post-Training Quantization (PTQ) framework tailored for multi-resolution video diffusion models (V-DMs).
• To address the limitations of existing PTQ methods that struggle with the multi-resolution nature of contemporary V-DMs.
• To enable faster inference and reduced model size for V-DMs without requiring retraining.

Main Methods:

• Proposing S²Q-VDiT⁺, a multi-resolution co-design PTQ framework integrating data, supervision, and quantizer strategies.
• Implementing Denoising-Prior Based Multi-Resolution Sampling for resolution-consistent latent construction and trajectory-aware policies.
• Utilizing Structure-Aware Multi-Resolution Distillation for enhanced structural alignment and Structure-Aware Multi-Resolution Distillation for spatial dependency transfer.
• Employing Debiased Modulated Quantization with asymmetric weight quantization and activation debiasing to mitigate distribution skew.

Main Results:

• S²Q-VDiT⁺ consistently outperforms existing PTQ baselines on state-of-the-art video generation models under W4A6 and W4A4 quantization.
• Achieved up to 2.08x end-to-end speedup in video generation inference.
• Reduced model storage by up to 3.8x and inference memory by up to 2.1x.

Conclusions:

• S²Q-VDiT⁺ offers an effective solution for compressing large video diffusion models, making them more deployable.
• The proposed multi-resolution co-design framework successfully addresses the challenges posed by the intrinsic multi-resolution nature of V-DMs.
• This work enables significant efficiency gains in terms of speed, storage, and memory for high-quality text-to-video generation.