Taking Language Embedded 3D Gaussian Splatting into the Wild

Area of Science:

• Computer Vision
• 3D Reconstruction
• Artificial Intelligence

Background:

• Large-scale photo collections enable 3D reconstruction for virtual exploration.
• Existing methods lack interactive semantic understanding of 3D scenes.
• Current scene understanding is limited to static text-image browsing.

Purpose of the Study:

• To develop an immersive 3D scene understanding approach using unconstrained photo collections.
• To extend 3D Gaussian splatting (3DGS) for open-vocabulary scene comprehension.
• To enable interactive semantic exploration beyond visual appearance.

Main Methods:

• Extended language-embedded 3DGS with multi-appearance CLIP features and uncertainty maps.
• Introduced a transient uncertainty-aware autoencoder and multi-appearance language field 3DGS.
• Developed a post-ensemble strategy for language feature fusion.
• Created the PT-OVS benchmark dataset for evaluation.

Main Results:

• Achieved state-of-the-art open-vocabulary segmentation on unconstrained photo collections.
• Demonstrated accurate semantic understanding beyond visual reconstruction.
• Outperformed existing methods in quantitative evaluations.

Conclusions:

• The proposed framework enables comprehensive 3D scene understanding from unconstrained photos.
• Facilitates interactive applications like open-vocabulary querying and 3D scene editing.
• Advances the field of semantic 3D reconstruction and virtual exploration.