Efficient Hybrid Hierarchical Clustering with Incremental Silhouette Score for Large, Noisy Datasets

Area of Science:

• Data Science
• Machine Learning
• Computational Statistics

Background:

• Hierarchical clustering analysis often faces computational challenges with large datasets.
• Evaluating cluster quality, such as with silhouette scores, can be computationally intensive.
• Existing algorithms may struggle with noisy data and accurate cluster number estimation.

Purpose of the Study:

• To introduce a novel framework for comprehensive clustering analysis.
• To develop an efficient incremental silhouette score calculation for hierarchical clustering.
• To enhance existing divisive partitioning algorithms for improved accuracy and robustness.

Main Methods:

• Developed an incremental silhouette score calculation reducing complexity from O(K N) to O(N).
• Enhanced the Principal Direction Divisive Partitioning (IPDDP) algorithm into PCA-MMDC and PCA-MMDC-sc.
• Integrated incremental silhouette calculation with enhanced algorithms into a hybrid clustering strategy.

Main Results:

• Achieved over a 100-fold speedup in silhouette evaluation for hierarchical clustering.
• Demonstrated significant improvements in clustering accuracy, with >10 percentage points ARI increase on noisy synthetic data vs. K-Means.
• Showcased effective management of noise and outliers in synthetic and real-world datasets.

Conclusions:

• The proposed framework offers a scalable and robust solution for complex clustering tasks.
• Incremental silhouette calculation makes large-scale hierarchical clustering analysis computationally feasible.
• Enhanced algorithms and hybrid strategy improve clustering accuracy and efficiency, particularly in noisy conditions.