SegGen: An Unreal Engine 5 Pipeline for Generating Multimodal Semantic Segmentation Datasets.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Data Science

Background:

• Collecting large-scale annotated datasets for semantic segmentation is costly and impractical.
• Existing synthetic data generation pipelines often lack multimodality, diversity, or require significant manual effort.
• Limitations in current methods hinder the development of robust semantic segmentation models.

Purpose of the Study:

• To present a fully automated pipeline for generating diverse, multimodal synthetic data for semantic segmentation.
• To overcome the limitations of manual data collection and constrained synthetic data generation methods.
• To enable high-throughput, scalable, and reproducible synthetic data generation with minimal human intervention.

Main Methods:

• Developed an automated pipeline within Unreal Engine 5 (UE5).
• Integrated UE5's procedural generation with a spline-following drone actor for data capture.
• Generated RGB and depth imagery with pixel-perfect semantic segmentation labels.

Main Results:

• Created a synthetic dataset of 1169 samples across two modalities and seven semantic classes.
• Demonstrated the pipeline's scalability and rapid environment variation capabilities.
• Trained benchmark computer vision models on the synthetic data, showing successful learning of semantic representations.

Conclusions:

• The automated UE5 pipeline effectively generates diverse, multimodal synthetic data for semantic segmentation.
• This approach accelerates research in multimodal perception and provides scalable, reproducible benchmarks.
• Game-engine-based synthetic data generation holds significant potential for advancing semantic segmentation models.