Unsupervised Tree Boosting for Learning Probability Distributions

Area of Science:

• Machine Learning
• Statistical Modeling
• Probability Theory

Background:

• Supervised tree boosting is a powerful ensemble method.
• Density estimation is crucial for understanding data distributions.
• Existing methods for unsupervised density estimation have limitations.

Purpose of the Study:

• To propose a novel unsupervised tree boosting algorithm for inferring sampling distributions.
• To introduce new concepts of distribution addition and residualization.
• To enable accurate multivariate density estimation without labeled data.

Main Methods:

• Developed an unsupervised tree boosting algorithm using additive tree ensembles.
• Introduced a new definition of multivariate cumulative distribution function (CDF) to enable distribution addition and residualization.
• Employed forward-stagewise fitting to minimize Kullback-Leibler divergence.
• Incorporated scale-dependent shrinkage and a two-stage marginal-copula fitting strategy.

Main Results:

• The algorithm successfully infers underlying sampling distributions from i.i.d. samples.
• Demonstrated the effectiveness of distribution addition and residualization for univariate and multivariate settings.
• Achieved competitive performance against state-of-the-art deep learning methods in multivariate density estimation.
• The algorithm provides analytic density evaluation and a generative model.

Conclusions:

• The proposed unsupervised tree boosting algorithm offers a novel approach to density estimation.
• The new multivariate CDF definition facilitates advanced distributional operations.
• This method shows promise for complex data analysis and generative modeling.