Enhancing 3D human pose estimation with NIR single-pixel imaging and time-of-flight technology: a deep learning...

Area of Science:

• Computer Vision
• Biomedical Imaging
• Machine Learning

Background:

• Extracting 3D human pose and shape from single images is challenging due to lighting and occlusions.
• Traditional RGB imaging methods struggle in adverse conditions.
• Single-pixel imaging (SPI) offers a robust alternative, especially in the near-infrared (NIR) spectrum.

Purpose of the Study:

• To investigate the use of NIR SPI with time-of-flight (TOF) for nighttime human detection.
• To develop a deep learning system for accurate 3D human pose and body shape extraction in low-light conditions.
• To evaluate the feasibility of NIR-SPI for outdoor surveillance and human sensing.

Main Methods:

• Utilized an SPI camera operating in the NIR spectrum (850-1550 nm) with TOF.
• Employed a vision transformers (ViT) model for human feature detection and extraction.
• Integrated extracted features with the SMPL-X 3D body model for deep learning-based 3D body shape regression.

Main Results:

• Demonstrated successful human detection in simulated nighttime environments using NIR-SPI.
• Achieved accurate 3D human pose and body shape reconstruction.
• Validated the potential of NIR-SPI as a reliable vision sensor for challenging conditions.

Conclusions:

• NIR-SPI with TOF is a promising technology for nighttime human detection and 3D pose/shape estimation.
• This approach overcomes limitations of traditional RGB imaging in low-light and occluded scenarios.
• The developed deep learning system shows significant potential for real-world applications in surveillance and human-computer interaction.