Variational Bayes Ensemble Learning Neural Networks With Compressed Feature Space | JoVE Visualize

Area of Science:

Machine Learning
Statistical Modeling
Computational Statistics

Background:

High-dimensional data (small n large p) presents challenges for traditional nonparametric classification.
Existing regularization techniques require training on the full dataset, leading to high computational and memory demands.
Random projection (RP) offers a computationally efficient alternative by compressing the feature space before neural network (NN) training.

Purpose of the Study:

To develop a robust nonparametric classification method for high-dimensional data.
To address the sensitivity of random projection methods to compression choices.
To improve prediction accuracy and provide uncertainty quantification in high-dimensional classification.

Main Methods:

Random projection (RP) of the high-dimensional feature space.
Training neural networks (NNs) on the compressed feature space.
Bayesian model averaging (BMA) to handle compression sensitivity and estimate intrinsic dimensionality.
A variational approach for simultaneous estimation of model weights and parameters, enabling parallel computation.
Asymptotic consistency analysis of the proposed algorithm.

Main Results:

The proposed method demonstrates improved prediction accuracy by averaging models near the intrinsic dimensionality.
The variational BMA approach offers computational gains comparable to frequentist methods while retaining Bayesian uncertainty quantification.
Asymptotic consistency of the algorithm is established under specific conditions for RPs and prior parameters.
Extensive numerical examples validate the empirical performance of the proposed method.

Conclusions:

The combination of random projections and Bayesian model averaging provides an effective solution for nonparametric classification in high-dimensional settings.
The variational approach enhances computational efficiency and scalability without sacrificing the benefits of Bayesian inference.
The method successfully quantifies uncertainty and identifies the intrinsic dimensionality of the feature space, leading to more reliable predictions.