Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Ordering and finding the best of K > 2 supervised learning algorithms.

Olcay Taner Yildiz1, Ethem Alpaydin

  • 1Department of Computer Engineering, Boğaziçi University, Istanbul, Turkey. yildizol@cmpe.boun.edu.tr

IEEE Transactions on Pattern Analysis and Machine Intelligence
|March 11, 2006
PubMed
Summary

We introduce the MultiTest algorithm to rank supervised learning algorithms by expected error, considering statistical tests and prior preferences. This method effectively identifies the best-performing algorithm from multiple options.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Continuously Constructive Deep Neural Networks.

IEEE transactions on neural networks and learning systems·2019
Same author

Tree Ensembles on the Induced Discrete Space.

IEEE transactions on neural networks and learning systems·2015
Same author

VC-dimension of univariate decision trees.

IEEE transactions on neural networks and learning systems·2015
Same author

Design and analysis of classifier learning experiments in bioinformatics: survey and case studies.

IEEE/ACM transactions on computational biology and bioinformatics·2012
Same author

Multiclass posterior probability support vector machines.

IEEE transactions on neural networks·2008
Same author

ANNSyS: an Analog Neural Network Synthesis System.

Neural networks : the official journal of the International Neural Network Society·2003

Area of Science:

  • Machine Learning
  • Statistical Learning Theory
  • Computational Statistics

Background:

  • Selecting optimal supervised learning algorithms is crucial for predictive modeling.
  • Existing statistical tests are limited to pairwise comparisons or checking for equality among multiple algorithms, failing to rank them by performance.
  • There is a need for a method that can rank multiple algorithms based on expected error, incorporating both empirical evidence and prior knowledge.

Purpose of the Study:

  • To propose a novel methodology, the MultiTest algorithm, for ordering supervised learning algorithms based on their expected error.
  • To develop a method that integrates results from pairwise statistical tests with prior preferences (e.g., algorithm complexity).
  • To provide a generalizable framework for ranking algorithms in both classification and regression tasks.

Related Experiment Videos

Main Methods:

  • The problem is framed in graph-theoretic terms.
  • A new algorithm, MultiTest, is proposed to order supervised learning algorithms.
  • The methodology combines pairwise statistical test outcomes with user-defined prior preferences.

Main Results:

  • Simulation results demonstrate the utility of the MultiTest algorithm.
  • The method was tested using five classification algorithms across 30 datasets.
  • The algorithm successfully orders algorithms based on expected error and prior preferences.

Conclusions:

  • The MultiTest algorithm offers an effective approach to ranking supervised learning algorithms.
  • The method addresses the limitations of existing pairwise and ANOVA tests for algorithm selection.
  • The proposed framework is adaptable for regression and other loss functions, enhancing its practical applicability.