Escaping The Curse of Dimensionality in Bayesian Model-Based Clustering

Area of Science:

• Statistics
• Machine Learning
• Computational Biology

Background:

• Bayesian mixture models are standard for clustering high-dimensional data.
• High dimensionality can lead to incorrect cluster counts in posterior inference.

Purpose of the Study:

• To explain the tendency of Bayesian clustering to produce too many or too few clusters in high dimensions.
• To propose a novel Bayesian clustering method that addresses high-dimensionality issues.

Main Methods:

• Analysis of the random partition posterior in a fixed-sample, increasing-dimension setting.
• Development of Latent Mixtures for Bayesian Clustering (Lamb) using low-dimensional latent variables.
• Scalable posterior inference techniques for the proposed model.

Main Results:

• Identified conditions for posterior inference favoring extreme clustering (all separate or all together) as dimension increases.
• Demonstrated that these conditions are prior-choice independent.
• Showcased Lamb's ability to avoid high-dimensionality pitfalls under mild assumptions.

Conclusions:

• The proposed Latent Mixtures for Bayesian Clustering (Lamb) offers a robust solution for high-dimensional data.
• Lamb demonstrates strong performance in simulations and real-world applications like single-cell RNA sequencing data analysis.