End-to-end Fusion3DGS: label-efficient multi-modal 3D instance segmentation based on Gaussian splatting

Accurate 3D instance segmentation is foundational for perception and decision making in embodied systems, yet prevailing approaches depend on densely annotated 3D point clouds, which are costly to acquire and difficult to scale across sensors and environments. We address this bottleneck with Fusion3DGS, an end-to-end, label efficient framework that couples 3D Gaussian Splatting with coordinated 2D-3D neural processing. From multi-view RGB images equipped only with 2D instance masks, our method optimizes a compact anisotropic Gaussian scene representation and performs instance reasoning via an occlusion aware cross-attention fusion stack. The weight-sharing lock imposes shape-consistent, gated coupling of early 2D and 3D kernels with conservative feedback and drift regularization to stabilize 2D-mask training and improve label efficiency, and a rendering consistency objective ties the Gaussian geometry to 2D supervision, enhancing boundary fidelity under occlusion and view changes. The ability to learn from widely available RGB data without dense 3D labels makes the approach practical for large-scale deployment.