Related Experiment Video
Updated: Jul 15, 2025
07:13
Author Spotlight: An Efficient and Robust Software for Automated Fusion of Multiple Preclinical Imaging Modalities
Published on: October 27, 2023
1.2K
Visual modalities-based multimodal fusion for surgical phase recognition
Bogyu Park1, Hyeongyu Chi1, Bokyung Park1
1AI Dev. Group, Hutom, Dokmak-ro 279, Mapo-gu, 04151, Seoul, Republic of Korea.
Computers in Biology and Medicine
|September 29, 2023
Summary
This study introduces a new method for surgical phase recognition using visual data and a novel visual kinematics-based index (VKI). This approach enhances understanding of surgical procedures and improves recognition accuracy, especially in complex surgeries.
Area of Science:
- Medical Robotics and Computer-Assisted Surgery
- Surgical Workflow Analysis
- Artificial Intelligence in Healthcare
Background:
- Surgical phase recognition is crucial for optimizing surgical workflows and resource allocation.
- Current methods often rely on limited information, such as instrument presence, hindering performance.
- A need exists for advanced techniques to improve the accuracy and robustness of surgical phase recognition.
Purpose of the Study:
- To propose a novel visual modality-based multimodal fusion method for enhanced surgical phase recognition.
- To introduce and validate a visual kinematics-based index (VKI) for better understanding of surgical procedures.
- To overcome the limitations of existing methods by incorporating instrument interaction dynamics.
Main Methods:
- Developed a multimodal fusion approach combining visual features with a visual kinematics-based index (VKI).
- Utilized a convolutional neural network (CNN)-based fusion technique for integrating visual data and VKI.
- Extracted VKI from surgical video data, capturing instrument movement and interrelations.
- Validated the methodology on both a virtual reality (VR) dataset (PETRAW) and a clinical distal gastrectomy dataset.
Main Results:
- The proposed fusion method significantly improved surgical phase recognition performance compared to CNN-based training alone.
- The visual kinematics-based index (VKI) provided complementary information, enhancing procedural understanding.
- Effective training results were achieved, outperforming fusion methods requiring extensive pre-trained data like Transformers.
- The approach demonstrated effectiveness in clinical settings, as validated on a distal gastrectomy dataset.
Conclusions:
- Visual modality-based multimodal fusion, incorporating VKI, offers a significant advancement in surgical phase recognition.
- The VKI effectively captures instrument interaction dynamics, improving the comprehension of surgical workflows.
- This method provides a robust and adaptable solution for surgical phase recognition across different surgical environments, including laparoscopic surgery.
