DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Current story visualization methods struggle with subject consistency in generated image sequences.
• Diffusion models show promise but lack coherence for multi-subject narratives.
• Automatic, open-domain story visualization remains a significant challenge.

Purpose of the Study:

• To introduce DreamStory, an automatic framework for generating coherent, multi-subject story visualizations.
• To address the challenge of maintaining subject consistency across image sequences derived from textual narratives.
• To develop a novel multi-subject consistent diffusion model for enhanced visual storytelling.

Main Methods:

• Leveraging large language models (LLMs) as story directors to generate descriptive prompts and annotate subjects.
• Utilizing a novel Multi-Subject consistent Diffusion model (MSD) incorporating Masked Mutual Self-Attention (MMSA) and Masked Mutual Cross-Attention (MMCA).
• Employing multimodal anchors (subject portraits and textual descriptions) to guide the MSD for consistent multi-subject generation.

Main Results:

• DreamStory successfully generates visually compelling and coherent story visualizations with consistent multi-subject representation.
• The MSD, with MMSA and MMCA, effectively ensures detailed appearance and semantic consistency of subjects.
• A new benchmark, DS-500, was established to evaluate story visualization framework performance, subject identification, and generation consistency.

Conclusions:

• DreamStory provides an effective solution for automatic, open-domain story visualization with improved multi-subject consistency.
• The proposed MSD architecture significantly enhances the quality and coherence of visual storytelling from text.
• The DS-500 benchmark facilitates future research and evaluation in the domain of story visualization.