PlaneRecTR++: Unified Query Learning for Joint 3D Planar Reconstruction and Pose Estimation.

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• 3D planar reconstruction involves complex sub-tasks like plane detection, segmentation, and pose estimation.
• Previous methods use a sequential, multi-stage approach, limiting performance by treating sub-tasks separately.
• Existing techniques often require external plane correspondence labeling and initial pose estimation.

Purpose of the Study:

• To develop a unified, single-stage framework for multi-view 3D planar reconstruction and pose estimation.
• To overcome the limitations of separate sub-task processing in existing methods.
• To eliminate the need for initial camera pose estimation and plane correspondence supervision.

Main Methods:

• Proposed PlaneRecTR++, a novel Transformer-based architecture.
• Integrated frame-wise plane detection, segmentation, parameter regression, and cross-frame correspondence into a single model.
• Employed a query-based learning approach for enhanced reasoning among semantic entities.

Main Results:

• Achieved mutual benefits across integrated sub-tasks through unified learning.
• Established a new state-of-the-art performance on benchmark datasets: ScanNetv1, ScanNetv2, NYUv2-Plane, and MatterPort3D.
• Demonstrated significant quantitative and qualitative improvements.

Conclusions:

• The unified single-stage framework effectively addresses multi-view 3D planar reconstruction and pose estimation.
• Integrating closely related sub-tasks leads to superior performance compared to sequential methods.
• PlaneRecTR++ offers a more efficient and accurate solution for 3D scene understanding.