Generalization analysis of deep CNNs under maximum correntropy criterion

Area of Science:

• Machine Learning
• Deep Learning
• Information Theory

Background:

• Convolutional Neural Networks (CNNs) are widely used but often rely on least squares loss, which is sensitive to noise and outliers.
• Robust regression methods are needed to enhance CNN performance in scenarios with heavy-tailed noise.

Purpose of the Study:

• To investigate the generalization error of deep CNNs with Rectified Linear Unit (ReLU) activation for robust regression.
• To explore the effectiveness of the maximum correntropy criterion (MCC) within an information-theoretic learning framework for CNNs.

Main Methods:

• Analyzing deep CNNs with ReLU activation functions.
• Employing the maximum correntropy criterion for empirical risk minimization.
• Investigating regression functions with additive ridge structure and noise with finite pth moments.
• Examining convergence rates in Sobolev spaces on the sphere.

Main Results:

• Deep CNNs with MCC achieve fast convergence rates for robust regression under specific conditions.
• These rates are comparable to the mini-max optimal rates of fully connected networks using Huber loss, with a logarithmic factor.
• Convergence rates are established for CNNs in Sobolev spaces on the sphere.

Conclusions:

• The maximum correntropy criterion enhances the robustness of deep CNNs in regression tasks, particularly with noisy data.
• This information-theoretic approach provides a viable alternative to traditional loss functions for improved generalization.
• The findings contribute to the theoretical understanding of deep learning for robust statistical estimation.