Minimizing Time Derivative of Loss for Efficient Generalization Enhancement With Applications to Nickel-Cobalt Alloy...

Area of Science:

• Machine Learning
• Artificial Intelligence
• Neural Network Optimization

Background:

• Deep neural networks (DNNs) exhibit limited generalization in complex, real-world scenarios.
• Flat minima in the loss landscape are crucial for enhancing DNN generalization.
• Existing methods for improving generalization often face challenges with training stability and convergence.

Purpose of the Study:

• To develop a novel optimization method for improving DNN generalization.
• To connect neural dynamics, specifically the time derivative of the gradient, with the identification of flat minima.
• To provide an effective training optimization solution for industrial defect detection tasks.

Main Methods:

• Applying neural dynamics to analyze the relationship between the gradient's time derivative and flat minima.
• Proposing a gradient-directed optimization method to steer network parameters towards flat minima during training.
• Conducting extensive experiments to validate the method's effectiveness and stability.

Main Results:

• The proposed method significantly enhances DNN generalization without compromising training stability or convergence.
• The optimizer markedly improves the generalization performance of YOLOv8n for nickel-cobalt alloy defect detection.
• Convergence proof is provided, demonstrating the method's feasibility.

Conclusions:

• Examining generalization through the lens of the loss's time derivative offers valuable insights.
• The gradient-directed optimization method presents a new paradigm for efficiently boosting DNN generalization capabilities.
• This research provides a practical solution for complex industrial defect detection using DNNs.