Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Ranks01:02

Ranks

277
Unlike parametric methods, nonparametric statistics are ideal for nominal and ordinal data, requiring fewer assumptions about the population's nature or distribution. This makes nonparametric methods easier to apply and interpret, as they do not depend on parameters like mean or standard deviation. One common approach in nonparametric analysis is to sort data according to a specific criterion. For instance, we might arrange weather data from hottest to coldest days in a month or rank cities...
277
Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving01:29

Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving

100
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
100
Pharmacokinetic Models: Comparison and Selection Criterion01:26

Pharmacokinetic Models: Comparison and Selection Criterion

134
Physiological and compartmental models are valuable tools used in studying biological systems. These models rely on differential equations to maintain mass balance within the system, ensuring an accurate representation of the dynamic processes at play.
Physiological models take a detailed approach by considering specific molecular processes. They can predict drug distribution, metabolism, and elimination changes, providing a comprehensive understanding of how drugs interact with the body.
134
Introduction to Nonparametric Statistics01:28

Introduction to Nonparametric Statistics

839
Nonparametric statistics offer a powerful alternative to traditional parametric methods, useful when assumptions about the population distribution cannot be made. Unlike parametric tests, which require data to follow a specific distribution with well-defined parameters (such as the mean and standard deviation), nonparametric tests do not require such constraints. This makes them particularly valuable when dealing with small sample sizes, skewed data, or ordinal and categorical variables.
One of...
839
Mechanistic Models: Compartment Models in Individual and Population Analysis01:23

Mechanistic Models: Compartment Models in Individual and Population Analysis

83
Mechanistic models are utilized in individual analysis using single-source data, but imperfections arise due to data collection errors, preventing perfect prediction of observed data. The mathematical equation involves known values (Xi), observed concentrations (Ci), measurement errors (εi), model parameters (ϕj), and the related function (ƒi) for i number of values. Different least-squares metrics quantify differences between predicted and observed values. The ordinary least...
83
Testing a Claim about Population Proportion01:24

Testing a Claim about Population Proportion

3.4K
A complete procedure for testing a claim about a population proportion is provided here.
There are two methods of testing a claim about a population proportion: (1) Using the sample proportion from the data where a binomial distribution is approximated to the normal distribution and (2) Using the binomial probabilities calculated from the data.
The first method uses normal distribution as an approximation to the binomial distribution. The requirements are as follows: sample size is large...
3.4K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Does the Depth of Hysteroscopic Endometrial Fundal Incision Influence Reproductive Outcomes in Oocyte Recipients? A Prospective Study.

Journal of clinical medicine·2026
Same author

GARCHNet: Value-at-Risk Forecasting with GARCH Models Based on Neural Networks.

Computational economics·2023
Same author

The Advantage of Case-Tailored Information Metrics for the Development of Predictive Models, Calculated Profit in Credit Scoring.

Entropy (Basel, Switzerland)·2022
Same author

Multiclass Image Classification Using GANs and CNN Based on Holes Drilled in Laminated Chipboard.

Sensors (Basel, Switzerland)·2021

Related Experiment Video

Updated: Aug 29, 2025

An R-Based Landscape Validation of a Competing Risk Model
05:37

An R-Based Landscape Validation of a Competing Risk Model

Published on: September 16, 2022

2.2K

A Probability-Based Models Ranking Approach: An Alternative Method of Machine-Learning Model Performance Assessment.

Stanisław Gajda1, Marcin Chlebus1

  • 1Faculty of Economic Sciences, University of Warsaw, Długa Street 44/50, 00-241 Warsaw, Poland.

Sensors (Basel, Switzerland)
|September 9, 2022
PubMed
Summary

The Probability-based Ranking Model Approach (PMRA) offers reliable machine learning model performance estimates, addressing weaknesses in methods like cross-validation. This new approach aids in selecting optimal models and hyperparameters.

Keywords:
Elo-based Predictive Powerhyperparameters tuningmachine learningmixed effects logistic regressionmodel performance assessmentmodel performance measuresmodel selection

More Related Videos

Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model
07:15

Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model

Published on: August 16, 2020

6.9K
Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances
07:35

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances

Published on: October 11, 2018

7.6K

Related Experiment Videos

Last Updated: Aug 29, 2025

An R-Based Landscape Validation of a Competing Risk Model
05:37

An R-Based Landscape Validation of a Competing Risk Model

Published on: September 16, 2022

2.2K
Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model
07:15

Machine Learning Algorithms for Early Detection of Bone Metastases in an Experimental Rat Model

Published on: August 16, 2020

6.9K
Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances
07:35

Selecting Multiple Biomarker Subsets with Similarly Effective Binary Classification Performances

Published on: October 11, 2018

7.6K

Area of Science:

  • Machine Learning
  • Statistical Modeling
  • Predictive Analytics

Background:

  • Classical performance measures for machine learning models have limitations, including difficulty in assessing performance differences and lack of interpretability.
  • Subsampling methods like bagging and cross-validation, while common, do not adequately address these limitations.
  • The Elo-based Predictive Power (EPP) was proposed to improve meta-measures of model performance but relies on potentially flawed assumptions.

Purpose of the Study:

  • To introduce the Probability-based Ranking Model Approach (PMRA) as a more reliable alternative for estimating machine learning model performance.
  • To address the limitations of existing performance estimation methods, particularly the inability to test significance and interpretability.
  • To provide a statistically sound method for comparing new algorithms against state-of-the-art models and optimizing hyperparameters.

Main Methods:

  • Developed the Probability-based Ranking Model Approach (PMRA), a modification of the EPP.
  • Utilized Mixed Effects Logistic Regression to calculate the probability of one model outperforming another.
  • Conducted empirical analysis on a real mortgage credits dataset comparing PMRA with k-fold cross-validation and EPP.

Main Results:

  • PMRA provides more reliable performance estimates compared to the original EPP.
  • Empirical analysis demonstrated PMRA's effectiveness in ranking 49 machine learning models on a mortgage credits dataset.
  • PMRA was successfully applied to hyperparameter tuning, offering a novel approach to selecting optimal configurations.

Conclusions:

  • PMRA offers a statistically robust framework for evaluating and comparing machine learning models.
  • The approach enhances the selection of optimal hyperparameters and guides further search in hyperparameter spaces.
  • PMRA enables the comparison of novel algorithms against existing state-of-the-art methods based on statistical criteria.