Simultaneous multimodal detection of hand acupoints and reflex zones for acupuncture robots

Acupuncture, a cornerstone of traditional Chinese medicine (TCM), faces challenges in standardization and precision, as its efficacy heavily relies on practitioner expertise. To address this, we propose a multimodal, multitask deep learning framework (MIMO-HAR) designed for intelligent acupuncture robots, enabling the simultaneous localization of hand acupoints and segmentation of reflex zones. Our approach uniquely integrates a vision transformer with TCM topological prior knowledge, enhancing both accuracy and interpretability. The system employs a dual-decoder architecture to process fused visual and coordinate data, concurrently outputting precise acupoint locations and high-fidelity segmentation masks. Evaluated on the public 11k Hands dataset, the MIMO-HAR framework demonstrates superior performance, achieving a mean intersection over union of 0.7267 for reflex zone segmentation and a root-mean-square error of 10.33 pixels for acupoint localization, significantly outperforming established baseline models. This study presents a robust and interpretable solution that advances the perceptual capabilities of intelligent acupuncture systems, laying a critical foundation for standardized and automated TCM therapies.