Average of Pruning: Improving Performance and Stability of Out-of-Distribution Detection

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Out-of-distribution (OOD) detection is crucial for safe neural network deployment in real-world scenarios.
• The stability and performance of OOD detection methods during training remain underexplored.
• Existing methods often exhibit unstable behavior, particularly as training nears completion or when training error is minimal.

Purpose of the Study:

• To investigate the instability of OOD detection performance during neural network training.
• To propose a novel method, Average of Pruning (AoP), to address these instabilities.
• To enhance the reliability and robustness of OOD detection systems.

Main Methods:

• Proposed Average of Pruning (AoP), integrating model averaging (MA) and pruning techniques.
• Model averaging (MA) was employed to smooth the optimization landscape and stabilize performance.
• Pruning was utilized to eliminate overfitting by removing redundant features, enhancing generalization.

Main Results:

• Demonstrated that OOD detection performance degrades with near-zero training error and exhibits sharp variations late in training.
• Showcased the effectiveness of AoP in mitigating training instability and improving OOD detection performance.
• Verified the method's efficacy across diverse datasets and neural network architectures.

Conclusions:

• Average of Pruning (AoP) effectively stabilizes and enhances out-of-distribution detection in neural networks.
• The combination of model averaging and pruning offers a robust solution to training-induced instabilities.
• AoP presents a promising approach for deploying more reliable AI systems in open-world environments.