InvPT++: Inverted Pyramid Multi-Task Transformer for Visual Scene Understanding.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Multi-task scene understanding models often process features locally, limiting global and cross-task interaction.
• This local processing hinders the full exploitation of task consistency in multi-task learning.

Purpose of the Study:

• To propose an Inverted Pyramid multi-task Transformer for effective global and cross-task interaction modeling.
• To enhance multi-task scene understanding by leveraging feature consistency across different scales.

Main Methods:

• Utilized a transformer encoder for task-generic feature extraction.
• Designed a transformer decoder with a novel UP-Transformer block for gradual resolution increase and cross-task interaction.
• Introduced Cross-Scale Self-Attention modules (Fusion Attention, Selective Attention) and an Encoder Feature Aggregation strategy.

Main Results:

• Demonstrated significant state-of-the-art performances on multiple 2D/3D multi-task benchmarks.
• Effectively modeled global context and cross-task interactions among spatial features.
• Showcased improved multi-scale information modeling in the decoder.

Conclusions:

• The proposed Inverted Pyramid multi-task Transformer effectively addresses limitations of local feature processing.
• The model achieves superior performance by enabling global, multi-scale, and cross-task interactions.
• This approach advances the field of multi-task scene understanding.