Related Experiment Videos
Relationship between Brier score and area under the binormal ROC curve
Mitsuru Ikeda1, Takeo Ishigaki, Kazunobu Yamauchi
1Department of Medical Information and Medical Records, Nagoya University Hospital, 65, Tsurumai-cho, Showa-ku, Nagoya 466-8560, Japan. a40495a@nucc.cc.nagoya-u.ac.jp
Computer Methods and Programs in Biomedicine
|February 21, 2002
Summary
This study explores the theoretical relationship between the Brier score (B) and the area under the receiver operating characteristic (ROC) curve (A(Z)) in signal detection theory. Findings reveal a monotonic decrease in Brier score as A(Z) increases, with implications for probability estimation.
Area of Science:
- Psychology
- Statistics
- Signal Detection Theory
Background:
- The Brier score (B) is a measure of the accuracy of probabilistic predictions.
- The receiver operating characteristic (ROC) curve, specifically the binormal ROC model, is used to evaluate the performance of diagnostic tests or decision-making processes.
- Understanding the relationship between B and the area under the ROC curve (A(Z)) can provide insights into the calibration and discriminative ability of probability estimates.
Purpose of the Study:
- To derive and investigate the theoretical functional relationship between the Brier score (B) and the area under the binormal receiver operating characteristic (ROC) curve (A(Z)).
- To explore how B varies with A(Z) under specific assumptions of signal detection theory, including perfect calibration.
Main Methods:
- Theoretical derivation of the functional relationship between B and A(Z) using parameters 'a' and 'b' from the binormal ROC model and the prior probability of signal events (alpha).
- Empirical investigation of the relationship by varying A(Z) while keeping parameters 'b' and 'alpha' constant.
Main Results:
- A theoretical functional relationship between the Brier score (B) and the area under the binormal ROC curve (A(Z)) was derived.
- Empirical findings indicate that for constant 'b' and 'alpha', B values monotonically decrease as A(Z) values increase.
- The relationship curves exhibit monotonically decreasing slopes, suggesting diminishing returns in Brier score improvement with increasing A(Z).
Conclusions:
- The study establishes a theoretical link between Brier score and ROC curve performance within signal detection theory.
- The findings suggest that improved discriminative ability (higher A(Z)) is associated with better probability calibration (lower B), up to a point.
- This relationship has implications for evaluating and interpreting probabilistic assessments in various fields utilizing signal detection theory.