Estimating the number of clusters via a corrected clustering instability

Area of Science:

• Cluster analysis
• Data mining
• Statistical modeling

Background:

• Determining the optimal number of clusters (k) is crucial for accurate cluster analysis.
• Existing instability-based methods can be influenced by cluster size distributions, limiting their effectiveness, particularly for large k.
• A need exists for robust methods to select k that are less sensitive to cluster size variations.

Purpose of the Study:

• To improve instability-based methods for selecting the number of clusters (k) in cluster analysis.
• To introduce a corrected clustering distance that mitigates the impact of cluster size distribution on instability.
• To compare the performance of model-based and model-free approaches for determining cluster instability.

Main Methods:

• Development of a corrected clustering distance metric.
• Evaluation of the corrected instability measure against existing methods across a range of k values.
• Comparative analysis of model-based and model-free cluster instability determination.

Main Results:

• The corrected instability measure demonstrates superior performance compared to current instability-based methods for all tested values of k.
• The proposed method effectively overcomes limitations of existing techniques, particularly for large k.
• Model-based and model-free approaches for determining cluster instability exhibit comparable performance.

Conclusions:

• The corrected clustering distance offers a more reliable approach for selecting the optimal number of clusters (k).
• This advancement enhances the robustness of instability-based methods in cluster analysis.
• The developed method is available as the R-package cstab for practical application.