A Validity Index for Prototype-Based Clustering of Data Sets With Complex Cluster Structures

Area of Science:

• Data Science
• Machine Learning
• Computer Science

Background:

• Evaluating unsupervised clustering results is challenging due to unknown data structures and cluster numbers.
• Existing cluster validity indices often fail with complex, non-spherical, or overlapping clusters, particularly in prototype-based clustering.
• Prototype-based clustering is crucial for large, high-dimensional datasets but requires effective validity assessment.

Purpose of the Study:

• To introduce a novel cluster validity index, Conn_Index, designed for prototype-based clustering.
• To address the limitations of existing indices in evaluating complex cluster structures.
• To provide a robust method for assessing the quality of clustering results.

Main Methods:

• Developed Conn_Index based on inter- and intra-cluster connectivities of prototypes.
• Utilized a 'connectivity matrix' derived from a weighted Delaunay graph to represent local data distribution.
• Applied the index to evaluate prototype-based clustering on synthetic and real-world datasets.

Main Results:

• Conn_Index demonstrated superior performance compared to existing validity indices in experiments.
• The new index effectively handles datasets with diverse cluster shapes, sizes, densities, and overlaps.
• Outperformed traditional indices in evaluating prototype-based clustering results on complex data.

Conclusions:

• Conn_Index offers a more reliable and accurate method for assessing prototype-based clustering quality.
• This new index is particularly valuable for analyzing large, high-dimensional datasets with intricate structures.
• The findings suggest Conn_Index as a significant advancement in unsupervised clustering evaluation.