On Perfect Clustering of High Dimension, Low Sample Size Data

Area of Science:

• Statistics
• Data Mining
• Machine Learning

Background:

• Traditional clustering algorithms struggle with high-dimensional, low-sample-size (HDLSS) data due to distance concentration and neighborhood structure issues.
• Euclidean distance-based methods are particularly affected, leading to poor performance in these scenarios.

Purpose of the Study:

• To introduce and evaluate a novel data-driven dissimilarity measure, MADD (Measure of Aggregated Dissimilarity Distance), designed to overcome HDLSS challenges in clustering.
• To demonstrate the effectiveness of MADD in improving clustering performance and cluster number estimation for high-dimensional datasets.

Main Methods:

• Development of the MADD dissimilarity measure, leveraging the distance concentration phenomenon in high dimensions.
• Theoretical and numerical studies to validate MADD's performance against traditional distance measures.
• Adaptation and creation of cluster number estimation algorithms, incorporating MADD for enhanced high-dimensional performance.
• Analysis of simulated and real-world datasets to showcase MADD's practical utility.

Main Results:

• Clustering algorithms utilizing MADD exhibit superior performance in high-dimensional, low-sample-size (HDLSS) settings compared to those using standard distance functions.
• MADD effectively addresses the adverse effects of distance concentration and neighborhood structure violations.
• Existing cluster number estimation algorithms show improved accuracy when integrated with MADD.
• A new, consistent estimator for the number of clusters in the HDLSS regime was developed and validated.

Conclusions:

• The MADD dissimilarity measure offers a robust solution for clustering high-dimensional data, particularly in HDLSS scenarios.
• MADD enhances the performance of both clustering algorithms and cluster number estimation techniques.
• This approach provides a valuable tool for effective cluster analysis in complex, high-dimensional datasets.