Rethinking Token-Wise Feature Caching: Accelerating Diffusion Transformers With Dual Feature Caching

Area of Science:

• Artificial Intelligence
• Computer Vision

Background:

• Diffusion Transformers (DiT) are powerful for image/video generation but computationally expensive.
• Feature caching methods accelerate DiTs by reusing past computations, skipping redundant steps.

Purpose of the Study:

• To critically evaluate the effectiveness of token-wise feature caching in DiTs.
• To investigate the necessity and importance of selected tokens in DiT computation.

Main Methods:

• Proposed Dual Feature Caching (DuCa) with iterative aggressive and conservative caching strategies.
• Employed random token selection for computation, challenging traditional importance-based methods.
• Evaluated DuCa on DiT, PixArt, FLUX, and OpenSora models.

Main Results:

• Demonstrated that consistently computing "important" tokens is unnecessary and often ineffective.
• Showed that random token selection can outperform traditional importance-based selection.
• Achieved significant improvements over previous token-wise feature caching methods.

Conclusions:

• Token-wise feature caching in DiTs is not as effective as previously assumed.
• DuCa offers a more efficient and effective approach to DiT acceleration through randomized, dual-strategy caching.