Enhancing human spatial awareness through augmented reality technologies

Area of Science:

• Computer Vision
• Robotics
• Human-Computer Interaction

Background:

• Augmented reality (AR) improves spatial perception by integrating digital information with the physical environment.
• Challenging environments like underwater and disaster zones limit direct human spatial awareness.
• Deep learning offers advanced methods for interpreting and completing 3D data.

Purpose of the Study:

• To review methodologies for AR-based spatial perception in challenging environments.
• To implement and test deep learning solutions within an AR system for 3D data interpretation and completion.
• To evaluate the system's effectiveness in recovering and augmenting spatial information.

Main Methods:

• Review of state-of-the-art deep learning approaches (voxel-based, point-based, view-based) for 3D data.
• Implementation of an AR system utilizing a robotic arm with an ultrasound sensor for underwater 2D scanning.
• 3D point cloud reconstruction, segmentation, and completion using deep learning networks.

Main Results:

• Experimental results demonstrate the system's capability to recover and augment unseen spatial information from synthetic and underwater data.
• Successful object identification through point cloud segmentation.
• Inference of missing structures via point cloud completion.

Conclusions:

• AR, coupled with robust sensing and 3D deep learning, significantly enhances spatial awareness in perception-limited environments.
• The study highlights the potential of AR for applications in underwater exploration and disaster response.
• Further research is needed for better point cloud metrics and more labeled sonar datasets.