Why You Cannot Rank First: Modifications for Benchmarking Six-Degree-of-Freedom Visual Localization Algorithms

Area of Science:

• Computer Vision
• Robotics
• Spatial Perception

Background:

• Visual localization faces challenges due to appearance variations from seasons and lighting.
• Existing benchmarks for long-term visual localization need improvement, particularly for autonomous vehicles.
• Advancements in technology have led to more datasets, promoting progress in 6-DOF visual localization.

Purpose of the Study:

• To rectify limitations in current public benchmarks for long-term visual localization.
• To enhance the evaluation algorithm's rationality and comprehensiveness for autonomous vehicle challenges.
• To ensure fair and accurate algorithmic performance evaluation.

Main Methods:

• Standardized preprocessing procedures including camera pose alignment and sequence information incorporation.
• Replacing individual camera positions with uniform vehicle poses for consistent evaluation.
• Introduction of a novel indicator to resolve ranking ties in the Schulze method.

Main Results:

• Proposed modifications significantly improve the evaluation of long-term visual localization.
• Simulations and experiments confirm the necessity and effectiveness of the amendments.
• Standardized procedures reduce human intervention and enhance result reliability.

Conclusions:

• The study provides a more robust and comprehensive benchmark for long-term visual localization.
• The proposed amendments are essential for accurate assessment of autonomous vehicle localization systems.
• The novel tie-breaking indicator ensures a clearer ranking of submitted methods.