Learning Spherical Convolution for 360^° Recognition.

Area of Science:

• Computer Vision
• Machine Learning
• 3D Graphics

Background:

• 360° cameras enable new applications in vision and augmented reality, but spherical images pose challenges for standard deep learning models.
• Existing methods for adapting Convolutional Neural Networks (CNNs) to spherical data often result in significant computational costs or reduced accuracy due to image distortion.

Purpose of the Study:

• To propose a novel Spherical Convolution Network (SphConv) for efficient and accurate visual recognition on 360° images.
• To enable the transfer of powerful pre-trained CNNs from perspective images to spherical image domains.
• To develop methods that mitigate distortion inherent in the equirectangular projection of spherical images.

Main Methods:

• Introduced Spherical Convolution Network (SphConv) to translate planar CNNs to the equirectangular projection of 360° images.
• Proposed two instantiations: Spherical Kernel (learning location-dependent kernels) and Kernel Transformer Network (KTN) (learning a functional transformation for kernels).
• Validated SphConv with multiple source CNNs and datasets, including a spherical Faster R-CNN for object detection without 360° annotations.

Main Results:

• SphConv effectively reproduces planar CNN filter outputs on 360° data, accounting for varying distortion.
• The Kernel Transformer Network (KTN) variant offers a lower memory footprint compared to Spherical Kernel.
• SphConv using KTN successfully preserves the accuracy of source CNNs while providing efficiency and transferability.

Conclusions:

• SphConv offers an efficient and accurate solution for visual recognition tasks using 360° imagery.
• The proposed method allows leveraging existing pre-trained CNNs for spherical data, enhancing scalability and performance.
• A spherical object detector was successfully trained using SphConv, demonstrating its utility without requiring 360° specific annotations.