ManlyMM-VAE: Deep Clustering With Manly Transformation Mixture Latent Embeddings for High-Dimensional Skewed Data

Clustering in high-dimensional data becomes particularly challenging when noise deviates from symmetric, zero-mean distributions such as the Gaussian or Laplace distribution. In these cases, asymmetric non-Gaussian distributions provide a more accurate representation of the data's structure, especially when learning low-dimensional subspaces. However, selecting an appropriate family of asymmetric distributions is difficult when variables within each group exhibit varying degrees of skewness-some being right-skewed and others left-skewed. In this study, we propose a novel deep generative clustering model, ManlyMM-VAE, based on the variational autoencoder (VAE) framework. The model incorporates the Manly transformation, a parametric technique that flexibly adjusts the skewness of each variable, effectively handling both right- and left-skewed data. A Manly transformation mixture model serves as the prior in the latent space. The evidence lower bound (ELBO) is derived using the stochastic gradient variational Bayes (SGVB) estimator to optimize model parameters. We also introduce a cross-validation procedure for model selection. Experimental results on both synthetic and real-world datasets show that ManlyMM-VAE effectively clusters data with varying skewness, achieving significantly higher accuracy than competing methods. Furthermore, its robustness is confirmed through clustering experiments on image data corrupted to various degrees.