Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understanding.

Area of Science:

• Computer Vision
• Deep Learning Architectures

Background:

• Image pyramids are crucial for multi-scale feature extraction in visual perception.
• Current methods incur high computational costs by using large models for all image resolutions.

Purpose of the Study:

• Introduce Parameter-Inverted Image Pyramid Networks (PIIP) to address computational inefficiency in multi-scale image processing.
• Develop a novel network architecture that balances computational cost and performance.

Main Methods:

• PIIP utilizes pretrained models (Vision Transformers or Convolutional Neural Networks) as network branches.
• Higher resolution images are processed by smaller network branches to optimize computation.
• A cross-branch feature interaction mechanism integrates information across different spatial scales.

Main Results:

• PIIP demonstrates superior performance over single-branch and existing multi-resolution approaches with reduced computational cost.
• Applied to InternViT-6B, PIIP improved detection and segmentation by 1-2% using 40-60% of original computation, achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K.
• PIIP-LLaVA achieved 73.0% accuracy on TextVQA and 74.5% on MMBench with limited training data.

Conclusions:

• PIIP offers an efficient solution for multi-scale visual perception and multimodal understanding.
• The proposed architecture effectively reduces computational load while maintaining or improving performance.
• PIIP shows significant promise for enhancing large-scale vision foundation models and multimodal large language models.