ISTR: Mask-Embedding-Based Instance Segmentation Transformer.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Transformer models show superior performance in instance-level recognition.
• Integrating mask embeddings with spatial information in transformer frameworks remains an underexplored area.

Purpose of the Study:

• To propose a novel transformer-based instance segmentation framework, the Instance Segmentation TRansformer (ISTR), incorporating Mask Meta-Embeddings (MME).
• To effectively combine mask embeddings and spatial information within a transformer architecture for enhanced instance segmentation.
• To improve the quality of mask embeddings through a mutual information maximization framework.

Main Methods:

• Developed ISTR with a recurrent refining head including Dynamic Box Predictor (DBP), Mask Information Generator (MIG), and Mask Meta-Decoder (MMD).
• Introduced Mask Meta-Embeddings (MME) interpreting mask encoding-decoding as a mutual information maximization problem, unifying schemes like PCA and DCT.
• Proposed a learnable Spatial Mask Tuner (SMT) to fuse spatial and embedding information from MIG.

Main Results:

• ISTR variants (ISTR-PCA, ISTR-DCT, ISTR-SMT) demonstrated effectiveness and efficiency in query-based instance segmentation.
• ISTR significantly outperformed existing mask-embedding-based models on the COCO dataset.
• ISTR achieved competitive performance against state-of-the-art models and strong baselines on COCO and Cityscapes datasets.

Conclusions:

• The proposed ISTR framework with MME effectively integrates mask embeddings and spatial information for transformer-based instance segmentation.
• ISTR offers a significant advancement over previous mask-embedding approaches and achieves state-of-the-art results.
• The method's flexibility and performance highlight the potential of meta-formulations for mask embedding optimization.