Method for user interface of large displays using arm pointing and finger counting gesture recognition

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Gesture Recognition

Background:

• Existing 3D pointing gesture recognition methods often neglect self-occlusion.
• Concurrent finger gesture recognition typically requires multiple sensors or cameras.
• Large display environments present unique challenges for gesture interaction.

Purpose of the Study:

• To propose a unified method for 3D pointing and finger gesture recognition in large displays.
• To address the challenge of self-occlusion in gesture recognition.
• To utilize a single Kinect device for both pointing and finger gesture recognition.

Main Methods:

• Employed a single Kinect device with a skeleton tracking model.
• Developed a self-occlusion compensation technique for shoulder position.
• Implemented a finger counting gesture recognition technique using depth images from the pointing vector's end.
• Utilized exception handling for self-occlusion scenarios.

Main Results:

• Significantly improved pointing accuracy, with an average root mean square error of approximately 13 pixels on a 1920 × 1080 screen.
• Achieved 98.3% accuracy for finger counting gesture recognition.
• Demonstrated effective handling of self-occlusion for enhanced gesture interpretation.

Conclusions:

• The proposed method effectively integrates pointing and finger gesture recognition using a single device.
• Self-occlusion compensation enhances the accuracy and robustness of 3D gesture recognition.
• This approach offers a practical solution for intuitive interaction in large display environments.