Pixels2Pose: Super-resolution time-of-flight imaging for 3D pose estimation

Area of Science:

• Computer Vision
• Sensor Technology
• Artificial Intelligence

Background:

• Single-photon-sensitive depth sensors are crucial for human pose and gesture recognition in advanced electronics.
• Current cost-effective sensors suffer from low spatial resolution, limiting their application scope.
• Existing limitations hinder detailed motion analysis and complex object detection.

Purpose of the Study:

• To significantly enhance the spatial resolution of low-resolution time-of-flight (ToF) sensors.
• To enable accurate three-dimensional human pose estimation from enhanced depth data.
• To develop an explainable framework for understanding the super-resolution process.

Main Methods:

• Implemented a temporal-to-spatial mapping technique to upscale sensor data.
• Utilized a novel explainable artificial intelligence framework for network analysis.
• Processed data from 4x4 pixel sensors to generate 32x32 pixel depth images.

Main Results:

• Achieved a drastic increase in depth image resolution from 4x4 to 32x32 pixels.
• Demonstrated accurate 3D human pose estimation for multiple individuals using the enhanced depth maps.
• Developed an interpretable model providing insights into data utilization and parameter relevance.

Conclusions:

• The developed super-resolution method significantly expands the utility of low-resolution ToF sensors.
• The approach shows potential for application in advanced human-computer interaction and robotics.
• This work paves the way for super-resolution techniques in other sensor modalities like radar and sonar.