Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Concept Videos

Probability Histograms01:17

Probability Histograms

12.8K
A probability histogram is a visual representation of a probability distribution. Similar a typical histogram, the probability histogram consists of contiguous (adjoining) boxes. It has both a horizontal axis and a vertical axis. The horizontal axis is labeled with what the data represents. The vertical axis is labeled with probability. Each rectangular bar in the histogram is 1 unit wide, which suggests that the area under each bar equals the probability, P(x), where x is 1, 2, 3, and so on.
12.8K
Contingency Table01:29

Contingency Table

3.3K
A contingency table provides a way of portraying data that can facilitate calculating probabilities. It is a method of displaying a frequency distribution as a table with rows and columns to show how two variables may be dependent (contingent) upon each other; The table helps determine conditional probabilities quite quickly and can help systematically organize, analyze and quantify data. The table displays sample values concerning two variables that may be dependent or contingent on one...
3.3K
Receiver Operating Characteristic Plot01:15

Receiver Operating Characteristic Plot

387
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
387
Determination of Expected Frequency01:08

Determination of Expected Frequency

2.4K
Suppose one wants to test independence between the two variables of a contingency table. The values in the table constitute the observed frequencies of the dataset. But how does one determine the expected frequency of the dataset? One of the important assumptions is that the two variables are independent, which means the variables do not influence each other. For independent variables, the statistical probability of any event involving both variables is calculated by multiplying the individual...
2.4K
Reliability and Validity01:29

Reliability and Validity

13.4K
Reliability and validity are two important considerations that must be made with any type of data collection. Reliability refers to the ability to consistently produce a given result. In the context of psychological research, this would mean that any instruments or tools used to collect data do so in consistent, reproducible ways.
13.4K
Binomial Probability Distribution01:15

Binomial Probability Distribution

14.5K
A binomial distribution is a probability distribution for a procedure with a fixed number of trials, where each trial can have only two outcomes.
The outcomes of a binomial experiment fit a binomial probability distribution. A statistical experiment can be classified as a binomial experiment if the following conditions are met:
There are a fixed number of trials. Think of trials as repetitions of an experiment. The letter n denotes the number of trials.
There are only two possible outcomes,...
14.5K

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Challenges of COVID-19 Case Forecasting in the US, 2020-2021.

PLoS computational biology·2024
Same author

Osband's principle for identification functions.

Statistical papers (Berlin, Germany)·2024
Same author

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States.

Proceedings of the National Academy of Sciences of the United States of America·2022
Same author

Evaluating epidemic forecasts in an interval format.

PLoS computational biology·2021
Same journal

In This Issue.

Proceedings of the National Academy of Sciences of the United States of America·2026
Same journal

Correction for Otsuki et al., Extracellular sulfatases support cartilage homeostasis by regulating BMP and FGF signaling pathways.

Proceedings of the National Academy of Sciences of the United States of America·2026
Same journal

Hive mind: Microbial communities and the making of memory.

Proceedings of the National Academy of Sciences of the United States of America·2026
Same journal

Targets for disease modification in schizophrenia: New findings add to evidence for the involvement of the immune complement system.

Proceedings of the National Academy of Sciences of the United States of America·2026
Same journal

Correction for Wang et al., The role of reduced aerosol masking from air pollutant emission reductions in recent global warming acceleration (2013-2023).

Proceedings of the National Academy of Sciences of the United States of America·2026
Same journal

Correction for Mishra, Ecology is not yet ready for AI-and why that matters.

Proceedings of the National Academy of Sciences of the United States of America·2026
See all related articles

Related Experiment Video

Updated: Nov 17, 2025

RBDT: A Computerized Task System based in Transposition for the Continuous Analysis of Relational Behavior Dynamics in Humans
11:09

RBDT: A Computerized Task System based in Transposition for the Continuous Analysis of Relational Behavior Dynamics in Humans

Published on: July 17, 2021

3.2K

Stable reliability diagrams for probabilistic classifiers.

Timo Dimitriadis1,2, Tilmann Gneiting2,3, Alexander I Jordan2

  • 1Alfred Weber Institute of Economics, Heidelberg University, 69115 Heidelberg, Germany; timo.dimitriadis@h-its.org.

Proceedings of the National Academy of Sciences of the United States of America
|February 18, 2021
PubMed
Summary
This summary is machine-generated.

We introduce the CORP approach for creating reliable probability forecasts. This method uses the pool-adjacent-violators (PAV) algorithm for statistically consistent and reproducible reliability diagrams, improving diagnostics in machine learning.

Keywords:
calibrationdiscrimination abilityprobability forecastscore decompositionweather prediction

More Related Videos

Generating Strictly Controlled Stimuli for Figure Recognition Experiments
05:39

Generating Strictly Controlled Stimuli for Figure Recognition Experiments

Published on: March 18, 2019

5.4K
A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

2.7K

Related Experiment Videos

Last Updated: Nov 17, 2025

RBDT: A Computerized Task System based in Transposition for the Continuous Analysis of Relational Behavior Dynamics in Humans
11:09

RBDT: A Computerized Task System based in Transposition for the Continuous Analysis of Relational Behavior Dynamics in Humans

Published on: July 17, 2021

3.2K
Generating Strictly Controlled Stimuli for Figure Recognition Experiments
05:39

Generating Strictly Controlled Stimuli for Figure Recognition Experiments

Published on: March 18, 2019

5.4K
A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments
08:12

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Published on: March 1, 2022

2.7K

Area of Science:

  • Statistics
  • Machine Learning
  • Data Science

Background:

  • Reliability diagrams assess forecast calibration by comparing predicted probabilities with observed frequencies.
  • Classical binning methods for reliability diagrams lack statistical consistency and reproducibility due to ad hoc decisions.

Purpose of the Study:

  • To introduce a novel, automated approach for generating statistically consistent and reproducible reliability diagrams.
  • To provide improved tools for the diagnostics and inference of probabilistic forecasts.

Main Methods:

  • The CORP approach utilizes nonparametric isotonic regression implemented via the pool-adjacent-violators (PAV) algorithm.
  • Reliability diagrams are generated by plotting PAV-(re)calibrated forecast probabilities.

Main Results:

  • The CORP approach produces provably statistically consistent and optimally binned reliability diagrams.
  • It allows for uncertainty quantification and provides a numerical measure of miscalibration.
  • A CORP-based Brier-score decomposition is introduced, generalizing to any proper scoring rule.

Conclusions:

  • The CORP approach offers a robust and reproducible method for evaluating probabilistic forecast reliability.
  • This method enhances diagnostic capabilities for a wide range of statistical and machine learning models.
  • The PAV algorithm provides a powerful tool for improving forecast calibration and inference.