Task-aware asynchronous multi-task model with class incremental contrastive learning for surgical scene...

Area of Science:

• Robotics
• Computer Vision
• Medical Informatics

Background:

• Robotic surgery scene understanding, including tool-tissue interaction recognition and automatic report generation, is crucial for intra-operative guidance and postoperative analysis.
• Domain shifts due to patient variation and new instruments degrade model performance in robotic surgery.
• Current methods often require multiple models, impacting computational efficiency and real-time performance.

Purpose of the Study:

• To propose a multi-task learning (MTL) model for simultaneous surgical report generation and tool-tissue interaction prediction.
• To address domain shift challenges in robotic surgery, including intensity shifts and novel instrument appearances.
• To enhance model learning and optimize the convergence of both tasks.

Main Methods:

• A multi-task learning (MTL) model with a shared feature extractor, a mesh-transformer branch for captioning, and a graph attention branch for interaction prediction was developed.
• Class incremental contrastive learning was employed in the shared feature extractor to handle domain shifts.
• Laplacian of Gaussian-based curriculum learning and task-aware asynchronous MTL optimization were integrated to improve learning and convergence.

Main Results:

• The MTL model achieved balanced performance with a BLEU score of 0.4049 for scene captioning and 0.3508 accuracy for interaction detection on the target domain.
• The model demonstrated performance on par with single-task models in domain adaptation scenarios.
• The proposed techniques effectively tackled intensity shifts and novel instrument appearances.

Conclusions:

• The developed multi-task model successfully adapted to domain shifts in robotic surgery.
• The model demonstrated the ability to incorporate novel instruments, performing tool-tissue interaction detection and report generation effectively.
• The MTL approach achieved comparable performance to single-task models while offering potential computational advantages.