DEGS: Deformable Event-Based 3D Gaussian Splatting From RGB and Event Stream

Area of Science:

• Computer Vision
• 3D Reconstruction
• Robotics

Background:

• Reconstructing dynamic 3D scenes from low-framerate videos is difficult due to motion uncertainty.
• Event cameras offer high-speed, motion-blur-robust data but lack color information.
• Integrating RGB and event data for 3DGS is challenging due to modality discrepancies.

Purpose of the Study:

• To develop a novel framework for jointly optimizing dynamic 3D Gaussian Splatting (3DGS) from RGB videos and event streams.
• To utilize event motion priors to guide the optimization of deformation fields in dynamic 3DGS.
• To overcome the challenges of integrating disparate RGB and event data modalities.

Main Methods:

• Extracting motion priors from event streams using LoCM unsupervised fine-tuning for scene adaptation.
• Developing a geometry-aware data association method for event-Gaussian motion correspondence.
• Employing motion decomposition and inter-frame pseudo-labeling strategies.

Main Results:

• The proposed framework successfully reconstructs dynamic 3DGS by jointly optimizing RGB and event data.
• Event motion priors significantly improve the accuracy and robustness of dynamic 3DGS.
• The method outperforms existing image-only and event-based approaches on synthetic and real-world data.

Conclusions:

• Combining low-framerate RGB videos with event streams offers a robust solution for dynamic 3DGS.
• The novel framework effectively leverages event data to enhance 3DGS reconstruction, especially in dynamic scenes.
• This approach advances the state-of-the-art in dynamic 3D scene reconstruction.