Capturing action triplet correlations for accurate surgical activity recognition.

Area of Science:

• Computer-assisted surgery
• Medical image analysis
• Surgical robotics

Background:

• Accurate surgical activity recognition is crucial for real-time decision support in computer-assisted surgery.
• Representing fine-grained surgical actions as triplets is key.
• A major challenge is capturing the correlations between these action triplets and their components.

Purpose of the Study:

• To propose a novel framework for accurate surgical activity recognition.
• To capture action triplet correlations at both feature and output levels.
• To enhance decision support in computer-assisted surgery systems.

Main Methods:

• Utilized a transformers-based spatial-temporal feature extractor with banded causal masks.
• Developed a graph convolutional networks (GCNs)-based module (TripletGCN) for feature enhancement by capturing triplet correlations.
• Implemented an adversarial learning process (TripletAL) to align joint triplet distributions for improved correlation.

Main Results:

• Achieved an average mean Average Precision (mAP) of 41.5% on the CholecT45 dataset.
• Attained an average mAP of 42.5% on the CholecT50 dataset.
• Demonstrated generalization capability for verb-target pair recognition on the SARAS-MESAD dataset.

Conclusions:

• The proposed framework effectively captures action triplet correlations for enhanced surgical activity recognition.
• The method shows significant performance improvements on public datasets.
• This approach holds promise for advancing real-time decision support in computer-assisted surgery.