Reliable Object Pose Alignment in Mixed-Reality Environments Using Background-Referenced 3D Reconstruction.

Area of Science:

• Computer Vision
• Mixed Reality Systems
• Robotics

Background:

• Mixed reality (MR) systems require accurate alignment between real-world objects and their virtual representations.
• Untracked object movements during inactive tracking periods cause pose inconsistencies, disrupting user interaction.
• Existing 3D reconstruction methods like MASt3R fail with moved objects as they assume static scenes.

Purpose of the Study:

• To develop a robust pipeline for correcting object pose misalignments in MR systems.
• To restore real-virtual consistency for moved objects after periods of inactive tracking.
• To enable seamless and consistent user interaction in dynamic MR environments.

Main Methods:

• A pipeline combining MASt3R's 3D outputs with a background-based alignment strategy.
• Segmentation of foreground and background to extract 3D background point sets for reference and current days.
• Estimation of affine transformation between background point sets for pose correction.

Main Results:

• The proposed method reliably corrects pose misalignments caused by untracked object movements.
• Significant improvement in real-virtual consistency compared to using MASt3R alone.
• Demonstrated effectiveness through experiments on real-world scenes.

Conclusions:

• The background-based alignment pipeline effectively recovers and applies true pose changes of moved objects.
• Restored real-virtual consistency enables consistent and improved user interaction in MR.
• The approach offers a robust solution for dynamic object handling in mixed reality.