Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

Area of Science:

• Artificial Intelligence
• Computer Vision
• Machine Learning

Background:

• Diffusion transformers (DiT) are a prominent architecture for image generation.
• Existing understanding of DiT performance factors remains limited.
• The adaLN-Zero conditioning mechanism shows superior performance over adaLN.

Purpose of the Study:

• To deeply investigate the factors contributing to the effectiveness of adaLN-Zero in DiT.
• To propose novel conditioning mechanisms and initialization strategies for DiT.
• To validate proposed methods through extensive experimentation.

Main Methods:

• Analysis of key components within adaLN-Zero: SE-like structure, zero-initialization, and update order.
• Development of an analysis-guided initialization strategy, adaLN-Gaussian.
• Introduction of an improved conditioning mechanism, SE-adaLN-Zero.
• Empirical validation on multiple datasets including ImageNet1K.

Main Results:

• Zero-initialization identified as the most critical factor for adaLN-Zero's performance.
• adaLN-Gaussian demonstrated consistent improvements in optimization efficiency.
• SE-adaLN-Zero showed effectiveness as an improved conditioning mechanism.
• Both methods generalized well across different image generation tasks.

Conclusions:

• The study provides crucial insights into DiT's conditioning mechanisms.
• adaLN-Gaussian and SE-adaLN-Zero offer practical enhancements for DiT performance and optimization.
• The findings contribute to advancing diffusion transformer architectures for image generation.