Method for Determining the Optimal Number of Clusters Based on Agglomerative Hierarchical Clustering

Area of Science:

• Data Science
• Machine Learning
• Statistics

Background:

• Accurate cluster number determination is vital for effective cluster analysis.
• Existing methods may not adequately capture geometric properties of sample clusters.
• Evaluating clustering quality requires robust validity indices.

Purpose of the Study:

• To introduce a novel clustering validity index based on sample geometry.
• To propose a method for determining the optimal number of clusters using agglomerative hierarchical clustering (AHC).
• To evaluate the proposed index and method across diverse dataset structures.

Main Methods:

• Definition of sample clustering dispersion and synthesis degrees.
• Design of a new clustering validity index.
• Implementation of a method for optimal cluster number selection with AHC.

Main Results:

• The proposed index effectively evaluates clustering results from AHC.
• The method successfully determines the optimal number of clusters for various data types (linear, manifold, annular, convex).
• Experimental validation confirms the index and method's performance.

Conclusions:

• The new geometric-based validity index offers a reliable approach to cluster analysis.
• The proposed method enhances the determination of the optimal number of clusters.
• This work provides a valuable tool for improving clustering quality across different data geometries.