How many clusters: a validation index for arbitrary-shaped clusters

Area of Science:

• Data Mining
• Machine Learning
• Bioinformatics

Background:

• Clustering validation indexes are crucial for assessing the quality of clustering results.
• Estimating the optimal number of clusters often relies heavily on these validation indexes.

Purpose of the Study:

• To introduce a new clustering validation index designed to detect arbitrary-shaped clusters.
• To integrate this novel index with the gap statistic for robust cluster number estimation.
• To evaluate the performance of the new method against existing validation techniques.

Main Methods:

• Development of a new validation index leveraging graph concepts and spatial data layout.
• Integration of the new index with the gap statistic framework.
• Comparative analysis using artificial datasets and gene expression data.

Main Results:

• The proposed method successfully identifies the correct number of arbitrary-shaped clusters across various scenarios.
• The new index demonstrates superior detection of underlying data structures compared to established methods.
• Results on gene expression data indicate the index's stability when subjected to data perturbations.

Conclusions:

• The novel graph-based validation index offers a more accurate approach to assessing clustering results, particularly for complex cluster shapes.
• The combination with the gap statistic provides a powerful tool for determining the optimal number of clusters.
• This method shows significant promise for applications in fields like bioinformatics and gene expression analysis.