Related Experiment Video
Updated: Dec 22, 2025
12:11
Computation of Atmospheric Concentrations of Molecular Clusters from ab initio Thermochemistry
Published on: April 8, 2020
8.6K
Probabilistic performance estimators for computational chemistry methods: Systematic improvement probability and
Pascal Pernot1, Andreas Savin2
1Institut de Chimie Physique, UMR8000, CNRS, Université Paris-Saclay, 91405 Orsay, France.
The Journal of Chemical Physics
|May 3, 2020
Summary
Evaluating computational chemistry theories requires new methods beyond standard error rankings. This study introduces a systematic improvement probability score and robust statistical indicators to assess ranking reliability, addressing benchmark dataset limitations.
Area of Science:
- Computational chemistry
- Theoretical chemistry
- Chemical physics
Background:
- Standard mean unsigned error rankings in computational chemistry are limited by non-normal error distributions and trends.
- Existing complementary statistics like error quantiles and prediction uncertainty partially address these limitations.
- Uncertainty arising from incomplete benchmark datasets is often overlooked in method evaluation.
Purpose of the Study:
- To introduce a novel scoring system, the systematic improvement probability, for direct system-wise comparison of absolute errors.
- To develop robust statistical indicators, inversion probability (P_inv) and ranking probability matrix (P_r), to quantify ranking uncertainty.
- To highlight the importance of correlations between error sets in statistical comparisons.
Main Methods:
- Direct system-wise comparison of absolute errors to calculate systematic improvement probability.
- Development of robust statistical indicators: inversion probability (P_inv) and ranking probability matrix (P_r).
- Analysis of correlations between error sets to assess their impact on ranking statistics.
Main Results:
- The proposed systematic improvement probability offers a new metric for evaluating computational chemistry methods.
- P_inv and P_r provide essential measures of ranking robustness against benchmark dataset incompleteness.
- Correlations between error sets significantly influence the reliability of statistical comparisons.
Conclusions:
- The standard mean unsigned error is insufficient for robustly ranking computational chemistry methods.
- Novel statistical approaches, including systematic improvement probability and robust indicators (P_inv, P_r), enhance the evaluation of theoretical models.
- Accurate assessment of computational chemistry theories necessitates considering error distributions, dataset incompleteness, and error correlations.
Related Concept Videos
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
226
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
226
01:05Estimation of the Physical Quantities
7.1K
On many occasions, physicists, other scientists, and engineers need to make estimates of a particular quantity. These are sometimes referred to as guesstimates, order-of-magnitude approximations, back-of-the-envelope calculations, or Fermi calculations. The physicist Enrico Fermi was famous for his ability to estimate various kinds of data with surprising precision. Estimating does not mean guessing a number or a formula at random. Instead, estimation means using prior experience and sound...
7.1K
00:59Uncertainty: Overview
1.4K
In analytical chemistry, we often perform repetitive measurements to detect and minimize inaccuracies caused by both determinate and indeterminate errors. Despite the cares we take, the presence of random errors means that repeated measurements almost never have exactly the same magnitude. The collective difference between these measurements - observed values - and the estimated or expected value is called uncertainty. Uncertainty is conventionally written after the estimated or expected value.
1.4K
01:10Propagation of Uncertainty from Systematic Error
1.2K
The atomic mass of an element varies due to the relative ratio of its isotopes. A sample's relative proportion of oxygen isotopes influences its average atomic mass. For instance, if we were to measure the atomic mass of oxygen from a sample, the mass would be a weighted average of the isotopic masses of oxygen in that sample. Since a single sample is not likely to perfectly reflect the true atomic mass of oxygen for all the molecules of oxygen on Earth, the mass we obtain from this...
1.2K
00:59Propagation of Uncertainty from Random Error
1.6K
An experiment often consists of more than a single step. In this case, measurements at each step give rise to uncertainty. Because the measurements occur in successive steps, the uncertainty in one step necessarily contributes to that in the subsequent step. As we perform statistical analysis on these types of experiments, we must learn to account for the propagation of uncertainty from one step to the next. The propagation of uncertainty depends on the type of arithmetic operation performed on...
1.6K
01:23Mechanistic Models: Compartment Models in Individual and Population Analysis
195
Mechanistic models are utilized in individual analysis using single-source data, but imperfections arise due to data collection errors, preventing perfect prediction of observed data. The mathematical equation involves known values (Xi), observed concentrations (Ci), measurement errors (εi), model parameters (ϕj), and the related function (ƒi) for i number of values. Different least-squares metrics quantify differences between predicted and observed values. The ordinary least...
195