3D Localization of Hand Acupoints Using Hand Geometry and Landmark Points Based on RGB-D CNN Fusion

Acupoint stimulation has proven to be of significant importance for rehabilitation and preventive therapy. Moxibustion, a kind of acupoint therapy, has mainly been performed by practitioners relying on manual localization and positioning of acupoints, leading to variance in the accuracy owing to human error. Developments in the automatic detection of acupoints using deep learning techniques have proven to somewhat tackle the problem. But the current methods lack depth-based localization and are thus confined to two-dimensional (2D) localization. In this research, a new approach towards 3D acupoint localization is introduced, based on a fusion of RGB and depth convolutional neural networks (CNN) to guide the manipulator. This research aims to tackle the challenge of real-time 3D acupoint localization in order to provide guidance for robot-controlled moxibustion. In the first step, the 3D sensor (Kinect v1) is calibrated and transformation matrix is computed to project the depth data into the RGB domain. Secondly, a fusion of RGB-CNN and depth-CNN is employed, in order to obtain 3D localization. Lastly, 3D coordinates are fed to the manipulator to perform artificially controlled moxibustion therapy. Furthermore, a 3D acupoint dataset consisting of RGB and depth images of hands, is constructed to train, validate and test the network. The network was able to localize 5 sets of acupoints with an average localization error of less than 0.09. Further experiments prove the efficacy of the approach and lay grounds for development of automatic moxibustion robots.