Marginalized neural network mixtures for large-scale regression

For regression tasks, traditional neural networks (NNs) have been superseded by gaussian processes, which provide probabilistic predictions (input-dependent error bars), improved accuracy, and virtually no overfitting. Due to their high computational cost, in scenarios with massive data sets, one has to resort to sparse gaussian processes, which strive to achieve similar performance with much smaller computational effort. In this context, we introduce a mixture of NNs with marginalized output weights that can both provide probabilistic predictions and improve on the performance of sparse gaussian processes, at the same computational cost. The effectiveness of this approach is shown experimentally on some representative large data sets.