Part-Level Car Parsing and Reconstruction in Single Street View Images

Area of Science:

• Computer Vision
• 3D Reconstruction
• Machine Learning

Background:

• Car parsing and reconstruction face challenges due to occlusions and viewpoint changes.
• Existing methods lack comprehensive car part information, limiting performance.
• Leveraging part-level details is crucial for robust automotive scene understanding.

Purpose of the Study:

• To propose the first part-aware approach for joint part-level car parsing and reconstruction in single street view images.
• To simultaneously estimate car pose, shape, and semantic parts without manual annotations on real images.
• To introduce a method for transferring part knowledge from synthetic to real data.

Main Methods:

• A novel network architecture incorporating dense part information for pose and shape estimation.
• A new 3D loss function for optimizing part-aware estimation.
• A class-consistent method for transferring part knowledge from synthesized to real images.
• Creation of a high-quality dataset with 348 car models, physical dimensions, and part annotations, generating 60K synthetic images.

Main Results:

• Effective transfer of part knowledge from synthetic to real images using the class-consistent method.
• Significant improvement in part segmentation performance on real street views.
• State-of-the-art performance in pose and shape estimation on the ApolloCar3D dataset.
• Outperforming previous methods by large margins in absolute and relative 3D car pose estimation (A3DP-Abs and A3DP-Rel).

Conclusions:

• Part information is highly beneficial for improving car parsing and reconstruction.
• The proposed part-aware approach effectively addresses limitations in existing methods.
• The developed dataset and transfer learning technique pave the way for future research in part-level car understanding.