InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis With Semantic Graph Prior.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Current 2D and 3D layout synthesis methods struggle with controllability due to implicit modeling of object distributions and relations.
• Generating realistic and controllable layouts from natural language instructions remains a significant challenge.

Purpose of the Study:

• To introduce InstructLayout, a novel generative framework designed to improve controllability and fidelity in 2D and 3D layout synthesis.
• To enable zero-shot versatility across downstream tasks using a semantic graph prior.
• To facilitate benchmarking of text-driven 2D and 3D scene synthesis.

Main Methods:

• Developed InstructLayout, integrating a semantic graph prior with a layout decoder.
• The semantic graph prior learns layout appearances and object distributions concurrently.
• Curated two high-quality datasets of layout-instruction pairs for benchmarking.

Main Results:

• InstructLayout demonstrates superior performance in both 2D and 3D layout synthesis tasks compared to state-of-the-art methods.
• The semantic graph prior enables zero-shot generalization to various downstream tasks.
• Experimental results confirm significant improvements in controllability and fidelity.

Conclusions:

• InstructLayout offers a powerful and versatile solution for text-driven 2D and 3D layout synthesis.
• The proposed semantic graph prior is effective in enhancing layout generation.
• The framework provides a strong baseline for future research in controllable scene synthesis.