Related Experiment Video
Updated: Oct 8, 2025

An R-Based Landscape Validation of a Competing Risk Model
Published on: September 16, 2022
Human Versus Machine: How Do We Know Who Is Winning? ROC Analysis for Comparing Human and Machine Performance under
Michael Merry1, Patricia Jean Riddle1, Jim Warren1
1School of Computer Science, University of Auckland, Auckland, New Zealand.
Background:
Receiver operating characteristic (ROC) analysis is commonly used for comparing models and humans; however, the exact analytical techniques vary and some are flawed.
Objectives:
The aim of the study is to identify common flaws in ROC analysis for human versus model performance, and address them.
Methods:
We review current use and identify common errors. We also review the ROC analysis literature for more appropriate techniques.
Results:
We identify concerns in three techniques: (1) using mean human sensitivity and specificity; (2) assuming humans can be approximated by ROCs; and (3) matching sensitivity and specificity. We identify a technique from Provost et al using dominance tables and cost-prevalence gradients that can be adapted to address these concerns.
Conclusion:
Dominance tables and cost-prevalence gradients provide far greater detail when comparing performances of models and humans, and address common failings in other approaches. This should be the standard method for such analyses moving forward.
More Related Videos
Related Concept Videos
Comparing the Survival Analysis of Two or More Groups
Receiver Operating Characteristic Plot
Mechanistic Models: Compartment Models in Individual and Population Analysis
Interpreting Run Charts
Bioequivalence Data: Statistical Interpretation
Statistical Analysis: Overview
One of the most commonly used statistical quantifiers is the mean, which is the ratio between the sum of the numerical values of all results and the...

