VisionHub: Learning Task-Plugins for Efficient Universal Vision Model

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Universal language models (NLP) have shown success, prompting research into unified frameworks for diverse visual tasks.
• Existing universal vision models struggle with adaptability, computational costs, workflow complexity, and performance limitations in diverse applications.
• Incomplete visual generation and perception capabilities hinder the generalizability of current models.

Purpose of the Study:

• To introduce VisionHub, a novel universal vision model designed for concurrent visual restoration and perception tasks.
• To enable streamlined transferability to downstream tasks with enhanced flexibility and efficiency.
• To address the limitations of existing models in terms of computational expense, workflow complexity, and performance versatility.

Main Methods:

• Leverages the frozen denoising U-Net architecture from Stable Diffusion as the core backbone.
• Incorporates lightweight task-plugins and a task router integrated onto the U-Net backbone for enhanced flexibility.
• Enables handling of various vision tasks via natural language instructions with minimal storage and operational overhead.

Main Results:

• VisionHub demonstrates efficiency and effectiveness across 11 different vision tasks.
• Achieves competitive performance on benchmarks, including 53.3% mIoU on ADE20K semantic segmentation.
• Shows strong results in depth estimation (0.253 RMSE on NYUv2) and pose estimation (74.2 AP on MS-COCO).

Conclusions:

• VisionHub offers a novel and efficient approach to universal vision modeling.
• The proposed architecture effectively manages multiple visual tasks and facilitates transfer learning.
• The model presents a promising solution for versatile and high-performance computer vision applications.