Sensor-Aware Recognition and Tracking for Wide-Area Augmented Reality on Mobile Phones

Area of Science:

• Computer Vision
• Mobile Augmented Reality
• Sensor Fusion

Background:

• Wide-area registration in outdoor mobile augmented reality (AR) is challenging due to scale variations and environmental complexities.
• Existing vision-based registration algorithms often suffer from fragility and drift, limiting their practical application.
• Robust and stable tracking is crucial for immersive outdoor AR experiences on mobile devices.

Purpose of the Study:

• To develop a sensor-aware system for large-scale outdoor augmented reality on mobile phones.
• To enhance the performance of recognition and tracking algorithms by integrating diverse sensor data.
• To improve the robustness and stability of AR registration in challenging outdoor environments.

Main Methods:

• Utilized a sensor-aware VLAD algorithm, adaptive to various scene scales, for complex scene recognition.
• Integrated GPS and gravity information to improve Visual-LiDAR Odometry (VLAD) performance.
• Employed an extended Kalman filter (EKF) to fuse data from inertial sensors and vision for improved tracking stability.

Main Results:

• Demonstrated significant enhancement in recognition rates for large-scale outdoor scenes.
• Achieved considerable improvements in tracking stability and robustness by fusing sensor and vision data.
• Effectively eliminated tracking jitters, leading to a smoother AR experience.

Conclusions:

• The proposed sensor-aware system offers a robust solution for wide-area registration in outdoor mobile AR.
• Sensor fusion, particularly with GPS and inertial data, significantly overcomes the limitations of vision-only approaches.
• The self-adaptive VLAD algorithm contributes to reliable recognition in complex and varied outdoor environments.