A clustering effectiveness measurement model based on merging similar clusters

Area of Science:

• Data Science
• Machine Learning
• Algorithm Analysis

Background:

• Affinity Propagation (AP) algorithm suffers from local clustering and inaccurate evaluation, especially with high cluster proportions.
• Existing internal evaluation indices lack variety, leading to invalid clustering results.

Purpose of the Study:

• To propose a clustering effectiveness measurement model that addresses AP algorithm limitations.
• To enhance clustering accuracy and provide reliable evaluation metrics.

Main Methods:

• Merging similar clusters based on inter-cluster similarity and average inter-cluster similarity to reduce the maximum number of clusters (K).
• Developing a new scheme to calculate intra-cluster compactness, inter-cluster relative density, and inter-cluster overlap coefficient.
• Designing internal evaluation indices based on intra-cluster cohesion and inter-cluster dispersion.

Main Results:

• The proposed model correctly performs clustering and classification on UCI and NSL-KDD datasets.
• Demonstrates accurate clustering range determination.
• Significantly outperforms three improved clustering algorithms in intrusion detection metrics like detection rate and false positive rate (FPR).

Conclusions:

• The developed model effectively overcomes AP algorithm's clustering and evaluation issues.
• Provides a robust framework for clustering and classification, particularly beneficial for intrusion detection systems.