Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generation

Area of Science:

• Artificial Intelligence
• Computer Vision

Background:

• Text-to-image models like Multimodal Diffusion Transformers (MMDiT) represent advanced AI.
• MMDiT reduces generation issues but struggles with prompts containing multiple similar subjects, causing subject neglect or mixing.
• Identified ambiguities include Inter-block, Text Encoder, and Semantic Ambiguity within the MMDiT architecture.

Purpose of the Study:

• To address subject neglect and mixing in MMDiT when generating multiple similar subjects.
• To propose novel techniques for on-the-fly latent repair during the denoising process.
• To enhance the accuracy and quality of text-to-image generation for complex prompts.

Main Methods:

• Proposed test-time optimization at early denoising steps to repair ambiguous latents.
• Introduced three loss functions: Block Alignment Loss, Text Encoder Alignment Loss, and Overlap Loss.
• Developed Overlap Online Detection and Back-to-Start Sampling Strategy to further mitigate semantic ambiguity.

Main Results:

• Experimental validation on a new dataset of similar subjects demonstrated superior generation quality.
• Achieved significantly higher success rates compared to existing text-to-image methods.
• Consistent improvements were observed across various MMDiT-based models like SD3, SD3.5, and FLUX.

Conclusions:

• The proposed method effectively resolves ambiguities in MMDiT for generating multiple similar subjects.
• The techniques offer general applicability and substantial improvements for text-to-image generation.
• This work advances the capabilities of diffusion transformer models in handling complex visual prompts.