Integrating Both Parallax and Latency Compensation into Video See-through Head-mounted Display.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Mixed Reality

Background:

• Achieving consistent spatial and temporal composition in mixed reality (MR) requires accurate reconstruction, occlusion, and reprojection of real and virtual scenes.
• Estimating dense depth maps for these tasks is computationally intensive, leading to unacceptable latency in head-mounted displays (HMDs).

Purpose of the Study:

• To develop a perspective-corrected video see-through MR HMD with edge-preserving occlusion and low-latency capabilities.
• To balance spatial consistency and low latency by rapidly generating depth maps focused on edge smoothness and disocclusion.

Main Methods:

• A hybrid edge refinement method using infrared masks and color-guided filters was employed.
• Disocclusions were filled using temporally cached depth maps.
• A two-phase temporal warping architecture synchronized camera pairs and displays for registration error reduction and head motion compensation.

Main Results:

• The system achieved low latency (less than 4 ms) relative to head motion.
• High spatial accuracy was demonstrated (less than 0.1° in size, less than 0.3° in position).
• A wearable prototype was implemented and evaluated through end-to-end measurements.

Conclusions:

• The proposed depth map generation method effectively balances spatial consistency and low latency for MR HMDs.
• This approach significantly enhances the realism of mixed reality systems.
• The system's performance metrics suggest a viable solution for next-generation MR devices.