A user-intent guided diffusion-segmentation collaborative framework for controllable digital media content generation

Area of Science:

• Artificial Intelligence
• Computer Vision
• Digital Media Creation

Background:

• Current user intent-guided digital media generation models face challenges with imprecise intent parsing, separate optimization of generation and segmentation, and limited robustness.
• Existing frameworks often struggle to effectively integrate user intent understanding with content creation and refinement processes.

Purpose of the Study:

• To propose the LLaDiSAM framework, an integrated solution for user intent-guided digital media content generation.
• To address limitations in intent parsing, generation-segmentation optimization, and model robustness.
• To establish an "understanding-generation-optimization" closed loop for enhanced media creation.

Main Methods:

• Developed the LLaDiSAM framework, combining LLaDA-V for intent parsing, DiT for diffusion generation, and FastSAM for segmentation optimization.
• Implemented a collaborative "understanding-generation-optimization" closed-loop system.
• Conducted experiments on four datasets, including LAION-5B and SA-1B, for performance evaluation.

Main Results:

• LLaDiSAM achieved high performance metrics: IFA of 0.89-0.92, FID as low as 6.67-7.21, and mIoU of 0.88-0.91.
• Demonstrated high robustness with coefficient of variation (CV) < 5% across 10 runs and a low deviation rate (7.5%) in challenging scenarios.
• Significantly outperformed 12 baseline models, including SDXL and DALL·E 3.

Conclusions:

• The LLaDiSAM framework offers a novel, efficient, and controllable paradigm for digital media creation.
• The study highlights the practical application potential of intent-guided generation technology.
• Future work will focus on improving multi-intent handling and segmentation module efficiency.