Gentle Nearest Neighbors Boosting over Proper Scoring Rules.

Area of Science:

• Machine Learning
• Computer Vision

Background:

• Nearest Neighbors algorithms require tailoring for boosting frameworks.
• Existing methods like UNN face numerical challenges impacting performance.
• Boosting algorithms, particularly Gentle Adaboost's lineage, lack formal convergence rates for certain approaches.

Purpose of the Study:

• To propose a novel, lightweight Newton-Raphson alternative for optimizing proper scoring rules in boosting.
• To establish formal convergence rates for this new method within the boosting framework.
• To compare the proposed method against existing state-of-the-art algorithms.

Main Methods:

• Developed a Newton-Raphson based algorithm (GNNB) for optimizing proper scoring rules.
• Established theoretical convergence rates for GNNB under the boosting framework.
• Conducted extensive experiments on diverse datasets, including computer vision databases (Caltech, SUN).

Main Results:

• GNNB demonstrates superior convergence rates and output quality compared to UNN.
• GNNB achieves performance comparable to or better than computationally intensive large margin methods.
• On large-scale datasets, GNNB proves a competitive alternative to stochastic gradient descent.

Conclusions:

• GNNB offers a significant advancement in tailoring nearest neighbors algorithms for boosting.
• The method provides a robust and efficient solution, outperforming previous approaches.
• GNNB represents a valuable new tool for machine learning practitioners, especially on large domains.