How the Outliers Influence the Quality of Clustering?

Area of Science:

• Data Science
• Machine Learning
• Cluster Analysis

Background:

• Clustering algorithms like Agglomerative Hierarchical Clustering (AHC) and K-Means are sensitive to data.
• Outliers can negatively impact the performance and efficiency of clustering algorithms.
• Assessing cluster quality requires robust validity indexes such as Davies-Bouldin and Dunn.

Purpose of the Study:

• To evaluate the impact of outliers on the performance of AHC and K-Means clustering algorithms.
• To compare the effectiveness of outlier detection algorithms (LOF, COF) in enhancing cluster quality.
• To determine if cluster quality improves after outlier removal.

Main Methods:

• Implemented and compared AHC and K-Means clustering algorithms.
• Utilized Local Outlier Factor (LOF) and Connectivity-based Outlier Factor (COF) for outlier detection.
• Removed varying percentages (1%, 5%, 10%) of detected outliers from real datasets.
• Assessed clustering quality using Davies-Bouldin and Dunn validity indexes.

Main Results:

• Outlier removal generally led to improved cluster validity scores across both algorithms.
• The degree of improvement varied depending on the dataset and the percentage of outliers removed.
• LOF and COF algorithms showed effectiveness in identifying outliers that impacted clustering performance.
• AHC and K-Means performance metrics showed enhancement after outlier exclusion.

Conclusions:

• The presence of outliers negatively affects the quality of clusters generated by AHC and K-Means.
• Proactive outlier detection and removal can significantly enhance the performance and reliability of clustering results.
• The choice of outlier detection method and the percentage of removal are critical factors for optimizing cluster analysis.