Towards Kilo-Hertz 6-DoF Visual Tracking Using an Egocentric Cluster of Rolling Shutter Cameras

Area of Science:

• Computer Vision
• Robotics
• Human-Computer Interaction

Background:

• Augmented reality (AR) requires precise device tracking for seamless integration of virtual and real worlds.
• Existing tracking systems often face limitations in accuracy, latency, or environmental adaptability.

Purpose of the Study:

• To develop a novel, high-frequency, low-latency 6-DOF head pose tracking method for AR applications.
• To leverage the capabilities of rolling shutter cameras for enhanced tracking performance.

Main Methods:

• Utilizing a cluster of rolling shutter cameras integrated onto the AR device.
• Employing the rapid row-by-row image capture of rolling shutter cameras as a high-frequency 1D sensor.
• Implementing markerless tracking in static indoor environments.

Main Results:

• Achieved tracking frequencies up to 56.7 kHz in synthetic data evaluations.
• Demonstrated markerless 6-DOF tracking with minimal latency.
• Validated accuracy against ground truth data using real-world prototype with commodity GoPro cameras.

Conclusions:

• The proposed rolling shutter camera-based tracking method offers a significant advancement in AR tracking frequency and performance.
• This approach enables more reliable and immersive AR experiences through accurate, low-latency head pose estimation.
• The system shows promise for generalized indoor environments and can be implemented with readily available hardware.