A Theoretical View of Linear Backpropagation and its Convergence

Area of Science:

• Deep learning
• Machine learning theory
• Neural network optimization

Background:

• Backpropagation (BP) is a standard algorithm for gradient computation in deep neural networks (DNNs), crucial for training and adversarial applications.
• A linear variant, LinBP, has shown empirical success in generating transferable adversarial examples for black-box attacks.
• Theoretical analysis and convergence properties of LinBP remain underexplored.

Purpose of the Study:

• To provide theoretical analysis for the Linear Backpropagation (LinBP) algorithm.
• To investigate LinBP's convergence properties in deep neural network training and adversarial attack scenarios.
• To complement and extend previous empirical findings on LinBP.

Main Methods:

• Theoretical analysis of LinBP's convergence in neural network learning tasks.
• Comparative study of LinBP against standard Backpropagation (BP) under identical hyper-parameter settings.
• Extensive experimental validation to confirm theoretical results.

Main Results:

• LinBP demonstrates faster convergence compared to standard BP in both adversarial attack generation and model training.
• Theoretical analysis supports the observed faster convergence of LinBP.
• Experimental results validate the theoretical findings across various tasks.

Conclusions:

• LinBP offers a potentially more efficient alternative to BP for specific deep learning applications.
• The theoretical underpinnings of LinBP's faster convergence are established.
• Further research into LinBP could lead to improved adversarial robustness and training efficiency.