Cluster Validity Index for Uncertain Data Based on a Probabilistic Distance Measure in Feature Space

Area of Science:

• Data Science
• Machine Learning
• Statistics

Background:

• Cluster validity indices (CVIs) are essential for determining the optimal number of clusters.
• Existing CVIs are primarily designed for 'certain data objects' lacking inherent uncertainty.
• Real-world data often contains uncertainty, posing limitations for traditional CVIs.

Purpose of the Study:

• To propose novel CVIs specifically designed for uncertain data.
• To address limitations of existing CVIs in handling arbitrary cluster shapes, sub-clusters, and noise.
• To improve the accuracy and robustness of cluster evaluation in the presence of data uncertainty.

Main Methods:

• Developed new CVIs utilizing kernel probabilistic distance measures.
• Transformed uncertain data into kernel spaces to calculate distances between distributions.
• Evaluated CVIs based on their ability to measure cluster compactness and separability in feature space.

Main Results:

• The proposed CVIs accurately measure cluster compactness and separability for arbitrary shapes.
• The new CVIs demonstrate robustness against noise and outliers within clusters.
• Evaluations on simulated and real-life uncertain data confirmed superior performance over existing methods.

Conclusions:

• The proposed kernel-based CVIs effectively evaluate clustering results for uncertain data.
• These novel indices offer improved accuracy and robustness compared to traditional methods.
• The findings suggest a significant advancement in handling uncertain data clustering problems.