Variational Nested Dropout

Area of Science:

• Machine Learning
• Deep Learning
• Probabilistic Modeling

Background:

• Nested dropout ranks network parameters or features by importance during training.
• Existing methods use fixed dropout rates, limiting adaptability in nested nets and generative models.
• Current approaches lack data-driven trajectory learning for performance decay and flexible representation learning.

Purpose of the Study:

• To develop a probabilistic counterpart to nested dropout for enhanced flexibility.
• To introduce Variational Nested Dropout (VND) for learning ordered representations and network architectures.
• To improve performance in classification and data generation tasks compared to existing nested dropout methods.

Main Methods:

• Proposed Variational Nested Dropout (VND) to sample multi-dimensional ordered masks efficiently.
• Developed a Bayesian nested neural network leveraging VND for learning parameter distribution order.
• Applied VND to generative models for learning ordered latent distributions.

Main Results:

• The proposed Bayesian nested neural network with VND outperformed standard nested networks in accuracy, calibration, and out-of-domain detection.
• VND-enhanced generative models showed superior performance in data generation tasks compared to related models.
• VND provides effective gradients for nested dropout parameters, enabling data-driven learning of feature importance.

Conclusions:

• Variational Nested Dropout offers a flexible and effective probabilistic approach to ordered representation learning and network architecture optimization.
• The proposed Bayesian nested neural network and generative models demonstrate significant improvements across various machine learning tasks.
• VND addresses limitations of fixed dropout rates, enabling more adaptive and data-driven model training.