Efficient High-Order Spatial Interactions for Visual Perception

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Vision Transformers (ViTs) achieve success via self-attention's spatial modeling.
• Convolutional Neural Networks (CNNs) are foundational in computer vision.
• Integrating ViT strengths into CNNs is an active research area.

Purpose of the Study:

• To introduce a convolution-based framework that replicates ViT's spatial modeling.
• To develop a novel operation, Recursive Gated Convolution (g nConv), for high-order spatial interactions.
• To create versatile backbones (HorNet, Hor3D, HorCLIP) for diverse visual tasks.

Main Methods:

• Proposed Recursive Gated Convolution (g nConv) for efficient, high-order spatial interactions.
• Developed generic vision backbones: HorNet (image recognition), Hor3D (point clouds), HorCLIP (vision-language).
• Integrated g nConv as a plug-and-play module into existing architectures.

Main Results:

• HorNet outperforms Swin Transformers and ConvNeXt on ImageNet, COCO, and ADE20K.
• g nConv improves dense prediction tasks with reduced computation.
• Hor3D shows efficacy in 3D semantic segmentation; HorCLIP excels in vision-language tasks.

Conclusions:

• g nConv effectively combines ViT and CNN merits, offering a new basic operation for visual modeling.
• The proposed HorNet family demonstrates strong performance and scalability.
• High-order spatial interactions via g nConv are beneficial across various visual modalities and tasks.