Enhancing classification accuracy in medical datasets using a hybrid distance and cluster refinement-based K-means...

Area of Science:

• Machine learning
• Data science
• Medical informatics

Background:

• Classic K-Means clustering struggles with medical data due to suboptimal distance metrics and lack of post-assignment refinement.
• These limitations lead to poor cluster cohesion and misgrouping in medical datasets.
• Existing methods often fail to fully capture complex medical data structures.

Purpose of the Study:

• To propose a novel enhanced K-Means clustering framework for medical data analysis.
• To address K-Means limitations by incorporating a hybrid distance metric and a cluster refinement mechanism.
• To improve the accuracy, interpretability, and robustness of clustering in healthcare applications.

Main Methods:

• Developed a hybrid distance approach combining cosine and cityblock (Manhattan) metrics with tunable weights.
• Implemented a cluster refinement mechanism using Z-score outlier detection for reassigning distant samples.
• Evaluated the framework on Breast Cancer Wisconsin (BCW) and Heart Disease datasets using multiple performance metrics.

Main Results:

• The enhanced K-Means achieved high accuracies: 0.9825 for BCW and 0.9000 for Heart Disease.
• Significantly outperformed traditional Euclidean and cosine-based K-Means methods.
• Demonstrated substantial improvements in homogeneity scores, indicating enhanced cluster quality and separation.

Conclusions:

• The proposed hybrid K-Means framework offers a practical and effective enhancement for medical data clustering.
• The combination of a hybrid distance metric and refinement step leads to superior performance over existing methods.
• This approach holds significant potential for improving unsupervised learning in medical data analysis and clinical decision-making.