Sparse clusterability: testing for cluster structure in high dimensions.

Area of Science:

• Statistics
• Data Mining
• Bioinformatics

Background:

• Cluster analysis is widely used to group data, assuming underlying cluster structures.
• Clusterability testing verifies this assumption, crucial for valid analysis.
• High-dimensional data presents unique challenges for traditional clustering methods.

Purpose of the Study:

• To develop and evaluate clusterability testing methods for high-dimensional data.
• To assess the performance of these methods using simulated and real-world datasets.
• To compare the proposed methods against existing techniques.

Main Methods:

• Utilizing sparse principal component analysis for clusterability testing.
• Evaluating Type I error and statistical power with simulated high-dimensional data.
• Applying methods to gene expression, microarray, and proteomics datasets.

Main Results:

• Proposed methods demonstrate reasonably low Type I error and maintained power across diverse datasets.
• Effectiveness varied with data structure and dimensionality; some datasets with close clusters were not detectable.
• This is the first comprehensive analysis of clusterability testing in high-dimensional settings.

Conclusions:

• The developed methods offer a viable approach for clusterability testing in high-dimensional data.
• The study highlights the importance of considering data structure and dimensionality.
• This work provides a foundation for future research in high-dimensional cluster analysis.