Learning Optical Flow and Scene Flow With Bidirectional Camera-LiDAR Fusion.

Area of Science:

• Computer Vision
• Robotics
• Machine Learning

Background:

• Estimating optical and scene flow from 2D and 3D data is crucial for autonomous systems.
• Existing methods struggle to effectively fuse multi-modal data, limiting performance.
• Early and late fusion strategies fail to fully leverage complementary sensor characteristics.

Purpose of the Study:

• To develop an end-to-end framework for joint optical and scene flow estimation.
• To improve the fusion of 2D image and 3D LiDAR data.
• To overcome limitations of existing fusion techniques in multi-modal perception.

Main Methods:

• A novel end-to-end framework with 2D and 3D branches and bidirectional fusion connections.
• Utilizing a point-based 3D branch to preserve LiDAR point cloud geometry.
• Introducing a learnable bidirectional camera-LiDAR fusion module (Bi-CLFM).
• Instantiating two fusion pipelines: CamLiPWC (pyramidal) and CamLiRAFT (recurrent).

Main Results:

• Both CamLiPWC and CamLiRAFT significantly outperform existing methods on the FlyingThings3D dataset.
• Achieved up to a 47.9% reduction in 3D end-point-error compared to prior best results.
• CamLiRAFT achieved state-of-the-art 4.26% error on the KITTI Scene Flow benchmark with fewer parameters.
• Demonstrated strong generalization and ability to handle non-rigid motion.

Conclusions:

• The proposed bidirectional fusion framework effectively integrates 2D and 3D data for superior scene flow estimation.
• The novel Bi-CLFM module enhances feature fusion between camera and LiDAR modalities.
• The CamLiRAFT model represents a new state-of-the-art in scene flow estimation, offering efficiency and robustness.