Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations

Area of Science:

• Computer Vision
• Machine Learning
• 3D Scene Reconstruction

Background:

• Dense 2D image feature extractors are crucial for visual analysis.
• Applying 2D extractors to multiple 3D reconstructible images presents challenges.
• Existing methods may require manual labels for semantic understanding in 3D scenes.

Purpose of the Study:

• To introduce Neural Feature Fusion Fields (N3F), a novel method for improving 2D image feature extractors in 3D scene analysis.
• To enable semantic understanding of 3D scenes using self-supervised learning, eliminating the need for manual labels.
• To demonstrate N3F's applicability to various neural rendering formulations and its performance improvements over 2D baselines.

Main Methods:

• N3F utilizes a pre-trained 2D image feature extractor as a teacher.
• A 3D student network, akin to a neural radiance field, is trained to distill features from the teacher network.
• The training process employs differentiable rendering techniques, integrating seamlessly with neural rendering frameworks like NeRF.

Main Results:

• N3F enables semantic understanding in scene-specific neural fields without manual annotation.
• The method consistently outperforms self-supervised 2D baselines across various tasks.
• Demonstrated improvements in 2D object retrieval, 3D segmentation, and scene editing using diverse video sequences.

Conclusions:

• N3F effectively bridges the gap between 2D feature extraction and 3D scene understanding.
• The proposed method offers a powerful, label-free approach for enhancing 3D scene analysis.
• N3F shows significant potential for applications in complex dynamic scenes and egocentric video analysis.