ImmerseGen: Agent-Guided Immersive World Generation with Alpha-Textured Proxies

Area of Science:

• Computer Graphics
• Virtual Reality
• Artificial Intelligence

Background:

• Automating immersive virtual reality (VR) scene creation is a significant challenge.
• Current methods often use complex geometry, leading to inefficient pipelines or reduced realism.
• There is a need for efficient and photorealistic VR world generation techniques.

Purpose of the Study:

• Introduce ImmerseGen, a novel agent-guided framework for compact and photorealistic VR world generation.
• Decouple realism from exhaustive geometric modeling for improved efficiency.
• Enable real-time rendering on mobile VR headsets.

Main Methods:

• Represent scenes as hierarchical compositions of lightweight geometric proxies with synthesized RGBA textures.
• Utilize terrain-conditioned texturing for base world generation and context-aware texturing for scenery.
• Employ vision-language model (VLM)-based agents for semantic analysis, asset placement, and multimodal enhancements (visual dynamics, ambient sound).

Main Results:

• ImmerseGen achieves superior photorealism and spatial coherence compared to existing methods.
• The framework demonstrates enhanced rendering efficiency, suitable for real-time mobile VR.
• Generated worlds are diverse and visually coherent.

Conclusions:

• ImmerseGen offers an efficient and effective solution for automated VR scene creation.
• The agent-guided framework successfully balances realism and computational efficiency.
• This approach advances the development of immersive and accessible VR experiences.