An Appearance-Semantic Descriptor with Coarse-to-Fine Matching for Robust VPR

Area of Science:

• Computer Vision
• Robotics
• Artificial Intelligence

Background:

• Semantic segmentation has advanced visual place recognition (VPR) using appearance-invariant semantic information.
• Challenges like semantic occlusion and sparsity in VPR limit reliance solely on semantic data for localization.
• Existing VPR algorithms struggle with complex urban environments featuring appearance and viewpoint variations.

Purpose of the Study:

• To propose a novel VPR framework, SemLook, that integrates semantic and appearance information for improved image matching accuracy and robustness.
• To address limitations of semantic-only VPR in scenarios with semantic occlusion and sparsity.
• To enhance real-time performance and accuracy in complex urban visual localization.

Main Methods:

• Developed a coarse-to-fine image matching strategy using novel SemLook descriptors.
• Constructed global SemLook descriptors from semantic contours for initial image screening, improving accuracy and real-time performance.
• Introduced local SemLook descriptors combining deep learning-extracted appearance features with semantic information for fine screening, addressing semantic overlap and sparsity.

Main Results:

• SemLook descriptors demonstrated superior performance compared to six state-of-the-art VPR algorithms across three public datasets (Extended-CMU Season, Robot-Car Seasons v2, SYNTHIA).
• Achieved 100% AUC on the Extended-CMU Season dataset and 99% AUC on the SYNTHIA dataset.
• The framework effectively handles complex image matching challenges in urban environments, showing enhanced accuracy and robustness.

Conclusions:

• The proposed coarse-to-fine strategy using global and local SemLook descriptors significantly improves VPR accuracy and robustness.
• Combining semantic and appearance information is crucial for effective localization in visually ambiguous or sparse environments.
• SemLook offers a promising solution for reliable visual place recognition in challenging real-world scenarios.