Related Experiment Video
Updated: Aug 20, 2025
Author Spotlight: Revolutionizing Remote Surgery with Augmented Reality and Robotics for Enhanced Precision and Accessibility
Published on: August 9, 2024
A virtual surgical prototype system based on gesture recognition for virtual surgical training in maxillofacial
Hanjiang Zhao1, Mengjia Cheng1, Jingyang Huang1
1Department of Oral and Cranio-Maxillofacial Surgery, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine; College of Stomatology, Shanghai Jiao Tong University; National Center for Stomatology; National Clinical Research Center for Oral Diseases; Shanghai Key Laboratory of Stomatology, Shanghai Research Institute of Stomatology, No. 639 Zhizaoju Road, Shanghai, 200011, China.
Virtual reality surgery training is enhanced with gesture recognition, improving realism and immersion. This system offers accurate hand gesture recognition and real-time feedback for authentic surgical simulation.
Area of Science:
- Biomedical Engineering
- Computer Science
- Surgical Simulation
Background:
- Virtual reality (VR) offers a promising platform for surgical training and education.
- Current VR surgical systems lack authenticity due to reliance on traditional input devices like mice or data gloves.
- Enhancing immersion in virtual surgery requires more intuitive and realistic interaction methods.
Purpose of the Study:
- To develop and evaluate a virtual surgery system integrating gesture recognition and real-time image feedback.
- To improve the authenticity and immersion of virtual surgical operations.
- To assess the feasibility of hand gesture recognition in a maxillofacial virtual surgical system.
Main Methods:
- Implemented an efficient, high-fidelity gesture recognition algorithm using hand data extraction and Support Vector Machine classification.
- Developed a collision detection algorithm utilizing Axis Aligned Bounding Box binary tree, Nominal Radius Theorem (NRT), and Separating Axis Theorem (SAT).
- Integrated these technologies into an existing maxillofacial virtual surgical system for evaluation.
Main Results:
- Achieved over 80% accuracy in recognizing ten static hand gestures, with some exceeding 90%.
- Ensured collision detection model generation met software requirements and achieved a gesture recognition response time under 40 ms (>25 Hz).
- Successfully demonstrated virtual surgical procedures like grabbing a scalpel, site selection, and incision with real-time feedback.
Conclusions:
- Integration of hand gesture recognition is a feasible approach to enhance interactivity and immersion in virtual surgical training.
- The developed system improves upon previous VR surgical platforms by incorporating advanced gesture recognition.
- This technology holds potential for more realistic and effective surgical education and practice.
