SIG-Former: monocular surgical instruction generation with transformers.

Area of Science:

• Computer Vision
• Natural Language Processing
• Surgical Technology

Background:

• Automatic surgical instruction generation is vital for intra-operative assistance.
• Challenges include complex surgical environments and the image-to-language gap.
• Existing methods struggle to accurately translate visual surgical data into coherent instructions.

Purpose of the Study:

• To introduce SIG-Former, a transformer-based network for generating surgical instructions from monocular RGB images.
• To address the challenges in understanding and translating surgical activities into human-like sentences.
• To improve intra-operative surgical assistance through automated instruction generation.

Main Methods:

• Utilizing a fine-tuned ResNet-101 for visual feature extraction.
• Employing transformer attention blocks to model visual representation, text embedding, and visual-textual relationships.
• Applying self-critical reinforcement learning to optimize the CIDEr score for sequence generation.

Main Results:

• Validation on the DAISI dataset, comprising 290 diverse clinical procedures.
• Demonstrated superior performance compared to baseline methods in quantitative and qualitative evaluations.
• SIG-Former effectively maps dependencies between visual features and textual information.

Conclusions:

• SIG-Former shows capability in linking visual and textual surgical information.
• Surgical instruction generation is an emerging field requiring further development.
• Future work includes expanding datasets, enhancing annotations, and pre-training models.