Development and evaluation of a novel framework to enhance k-NN algorithm's accuracy in data sparsity contexts

Area of Science:

• Machine Learning
• Data Science
• Artificial Intelligence

Background:

• The k-Nearest Neighbors (k-NN) algorithm faces accuracy challenges in sparse data environments.
• Existing k-NN implementations often struggle with suboptimal training processes and data structure limitations.

Purpose of the Study:

• To present a novel framework for implementing the k-NN algorithm to enhance accuracy in sparse data contexts.
• To optimize data structures and reduce implementation uncertainty within the k-NN algorithm.

Main Methods:

• Developed a framework utilizing composite datasets generated via a data-driven fuzzy Analytic Hierarchy Process (AHP) weighting scheme.
• Enhanced informational content of initial datasets to reduce entropy.
• Evaluated the framework on 75 public and 3 generated datasets.
• Assessed sensitivity using various distance metrics and generalized findings with non-parametric hypothesis tests.

Main Results:

• Demonstrated significant improvements in k-NN algorithm accuracy across various k-parameter values.
• The composite data structures enhanced informational content, leading to accuracy gains and improved scalability.
• The framework proved effective particularly in data-sparse contexts.

Conclusions:

• The novel framework effectively enhances k-NN accuracy and scalability in sparse data settings.
• Reducing data entropy through optimized composite structures is critical for improving explainable machine learning performance.
• The approach offers a valuable and interpretable tool for data structure transformation in sparse environments.