End-to-End Multitask Learning With Vision Transformer.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Multitask learning (MTL) in computer vision (CV) faces challenges with parameter sharing and network design.
• Existing deep MTL models are often vulnerable to under-constrained parameters, limiting performance.
• Vision Transformer (ViT) architectures have shown recent success, offering new possibilities for representation learning.

Purpose of the Study:

• To propose a novel multitask representation learning method for computer vision using the Vision Transformer (ViT) architecture.
• To develop an efficient MTL model that addresses the limitations of existing methods, particularly with less related tasks.
• To evaluate the proposed method's performance against current state-of-the-art approaches on benchmark datasets.

Main Methods:

• Introduced Multitask Vision Transformer (MTViT), a model featuring multiple transformer branches for sequential processing of image patches (tokens) across tasks.
• Proposed a cross-task attention (CA) module enabling task tokens to exchange information between branches.
• Leveraged ViT's self-attention mechanism for intrinsic feature extraction, achieving linear time complexity for memory and computation.

Main Results:

• MTViT demonstrated performance on par with or exceeding existing Convolutional Neural Network (CNN)-based MTL methods on NYU-Depth V2 and CityScapes datasets.
• Experiments on a synthetic dataset revealed that MTViT performs exceptionally well even when tasks have low relatedness.
• The method achieved linear time complexity, a significant improvement over the quadratic complexity of some prior models.

Conclusions:

• MTViT offers an effective and efficient solution for multitask representation learning in computer vision.
• The proposed architecture, particularly the cross-task attention module, enhances performance, especially in scenarios with diverse or less related tasks.
• MTViT represents a promising advancement in deep MTL, leveraging the strengths of Vision Transformers for improved performance and efficiency.