Joint-Confidence-Guided Multi-Task Learning for 3D Reconstruction and Understanding From Monocular Camera.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Monocular 3D reconstruction is crucial for computer vision tasks.
• Current multi-task learning methods have limitations in incorporating spatial-aware loss information.
• Accurate geometric and semantic understanding from single images remains a challenge.

Purpose of the Study:

• To propose a novel Joint-confidence-guided network (JCNet) for simultaneous multi-task prediction.
• To improve the integration of spatial-aware information into loss functions for 3D understanding.
• To achieve state-of-the-art performance in geometric-semantic prediction and uncertainty estimation.

Main Methods:

• Developed JCNet, a network that simultaneously predicts depth, semantic labels, surface normal, and a joint confidence map.
• Introduced a Joint Confidence Fusion and Refinement (JCFR) module for multi-task feature fusion.
• Implemented a Stochastic Trust Mechanism (STM) to balance training attention and a calibrating operation to prevent overfitting.

Main Results:

• JCNet achieved state-of-the-art performance on the NYU-Depth V2 and Cityscapes datasets.
• The proposed methods demonstrated superior geometric-semantic prediction accuracy.
• Effective uncertainty estimation capabilities were achieved through the joint confidence map.

Conclusions:

• JCNet effectively addresses limitations in spatial-aware information for monocular 3D reconstruction.
• The joint confidence-guided approach enhances multi-task learning for computer vision.
• The model provides robust performance in both prediction accuracy and uncertainty estimation.