K Important Neighbors: A Novel Approach to Binary Classification in High Dimensional Data

Area of Science:

• Machine Learning
• Statistical Classification
• High-Dimensional Data Analysis

Background:

• K nearest neighbors (KNN) classifiers are simple but struggle with high-dimensional data due to nuisance features.
• The curse of dimensionality significantly impacts KNN accuracy in complex datasets.
• Feature selection and robust distance metrics are crucial for effective classification in high dimensions.

Purpose of the Study:

• To propose K important neighbors (KIN) as a novel approach for binary classification in high-dimensional problems.
• To address the limitations of traditional KNN in high-dimensional settings.
• To develop a method that effectively reduces dimensionality and improves classification accuracy.

Main Methods:

• Implemented smoothly clipped absolute deviation (SCAD) logistic regression for initial feature selection.
• Developed a hybrid dissimilarity measure combining SCAD coefficients and Euclidean distance.
• Incorporated feature importance derived from SCAD into the distance calculation for KNN.

Main Results:

• KIN demonstrated superior performance compared to standard KNN in simulation studies.
• The method achieved significant dimension reduction by effectively eliminating noninformative features.
• KIN showed competitive or superior accuracy, especially in very sparse high-dimensional settings, outperforming SVM and Random Forest.

Conclusions:

• KIN offers an effective solution for high-dimensional binary classification by integrating feature importance into a KNN framework.
• The SCAD-driven dissimilarity measure successfully mitigates the curse of dimensionality.
• KIN provides a robust and accurate alternative to existing classification methods for complex, high-dimensional datasets.