Smart Task Assistance in Mixed Reality for Astronauts.

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Aerospace Engineering

Background:

• Mixed reality (MR) enhances astronaut training by integrating virtual information with real environments.
• Current MR systems struggle with dynamic scenes due to reliance on spatial anchors for static object alignment.
• A novel approach is needed to enable virtual-real fusion with movable objects in training simulations.

Purpose of the Study:

• To develop a smart task assistance method for astronaut training that overcomes limitations of static spatial anchors.
• To enable accurate pose estimation and virtual-real fusion for both fixed and movable objects in mixed reality environments.
• To enhance the practicality and applicability of mixed reality in astronaut training scenarios.

Main Methods:

• Object detection using YOLOv5s to identify and segment partial point clouds of both fixed and movable objects.
• Pose estimation via iterative closest point (ICP) algorithm aligning partial point clouds with template point clouds.
• Virtual-real fusion executed without dependence on preset spatial anchors or background information.

Main Results:

• Automatic pose estimation for both fixed and movable objects was achieved.
• The proposed method successfully performed virtual-real fusion without requiring background data or predefined spatial anchors.
• Experimental results demonstrated the system's capability to handle dynamic objects in mixed reality.

Conclusions:

• The developed method offers a practical solution for integrating movable objects into mixed reality astronaut training.
• This approach expands the potential applications of MR in complex training simulations by enabling dynamic object interaction.
• Volunteer feedback indicated high practicality, suggesting significant potential for improving astronaut skill acquisition.