Towards a Deeper Understanding of Global Covariance Pooling in Deep Learning: An Optimization Perspective

Area of Science:

• Computer Vision
• Deep Learning Optimization

Background:

• Global covariance pooling (GCP) is a promising alternative to global average pooling for enhancing deep convolutional neural networks (CNNs).
• The precise mechanisms by which GCP, particularly its post-normalization, impacts deep learning optimization remain incompletely understood.

Purpose of the Study:

• To investigate the optimization effects of Global Covariance Pooling (GCP) in deep learning architectures.
• To analyze the role of post-normalization in GCP and propose an improved normalization technique.

Main Methods:

• Analysis of GCP with matrix power normalization on optimization loss and gradient computation.
• Exploration of post-normalization effects on GCP from an optimization standpoint.
• Proposal of a novel normalization method, DropCov.

Main Results:

• GCP enhances Lipschitzness of the optimization loss and promotes flatter local minima.
• GCP improves gradient predictiveness and acts as a preconditioner for gradients.
• The proposed DropCov normalization further refines GCP's optimization benefits.

Conclusions:

• GCP offers significant advantages in deep learning, including faster convergence, enhanced model robustness, and improved generalization.
• The findings provide a deeper understanding of GCP's optimization behavior and introduce an effective normalization strategy.