Hand and Object Segmentation from Depth Image using Fully Convolutional Network.

Area of Science:

• Computer Vision
• Robotics
• Biomedical Engineering

Background:

• Semantic segmentation is vital for accurate hand and object tracking in applications like upper limb rehabilitation.
• Existing segmentation methods lack sufficient semantic labels and large-scale datasets for training deep neural networks.
• The absence of comprehensive datasets hinders the development of robust, end-to-end deep learning models for hand-object interaction.

Purpose of the Study:

• To develop a framework for generating a synthetic dataset for hand and object segmentation in upper limb rehabilitation.
• To train a deep neural network using this synthetic dataset for improved segmentation accuracy.
• To enable real-time hand-object interaction tracking for rehabilitation applications.

Main Methods:

• A novel framework was created to generate a large-scale synthetic dataset with rich semantic labels, focusing on hand-object interactions.
• A deep neural network was proposed and trained on the generated synthetic depth images.
• The trained network was evaluated on real-world depth images to assess its performance and inference speed.

Main Results:

• The proposed deep neural network achieved a mean intersection over union (mIoU) of 70.4% when tested on real depth images.
• Despite being trained on synthetic data, the network demonstrated strong generalization capabilities to real-world scenarios.
• The network's inference time was approximately 6 milliseconds on a GPU, indicating its suitability for real-time applications.

Conclusions:

• The developed synthetic dataset and deep neural network framework effectively address the limitations of current hand and object segmentation methods.
• The approach enables accurate, real-time semantic segmentation for hand-object interactions, particularly beneficial for upper limb rehabilitation robotics.
• This work paves the way for more sophisticated and responsive robotic assistance in therapeutic settings.