Generalized Few-Shot MM-Former For Surgical Scene Panoptic Segmentation.

Area of Science:

• Computer Vision
• Medical Imaging
• Artificial Intelligence

Background:

• Panoptic segmentation is vital for surgical scene understanding but faces challenges due to high annotation costs and class imbalance.
• Limited data for specific surgical categories hinders the performance of existing models.

Purpose of the Study:

• To develop a generalized few-shot learning framework for accurate panoptic segmentation in surgical scenes.
• To address the challenge of limited annotated data in surgical datasets.

Main Methods:

• A three-stage framework was proposed, starting with fine-tuning a stable diffusion model on surgical image-text pairs for multi-scale representations.
• A Mask2Former-based decoder was trained on base classes, generating mask proposals.
• A novel N-to-M mask matching method was introduced to identify novel classes using limited samples.

Main Results:

• The proposed MM-former achieved outstanding results on the newly built CholecPanSeg dataset under limited data conditions.
• The method demonstrated superior performance compared to previous approaches in few-shot surgical panoptic segmentation.
• Accurate identification of novel class objects was achieved in a single pass.

Conclusions:

• The generalized few-shot MM-former effectively handles class imbalance and limited data in surgical panoptic segmentation.
• The proposed N-to-M mask matching method enables robust identification of rare surgical categories.
• This framework significantly advances surgical scene understanding and paves the way for improved AI-assisted surgery.