Related Experiment Video
Updated: Jun 15, 2026
07:43
A Naturalistic Setup for Presenting Real People and Live Actions in Experimental Psychology and Cognitive Neuroscience Studies
Published on: August 4, 2023
Is an ROC-type response truly always better than a binary response in observer performance studies?
David Gur1, Andriy I Bandos, Howard E Rockette
1University of Pittsburgh, Department of Radiology, Radiology Imaging Research, Pittsburgh, PA 15213, USA. gurd@upmc.edu
Academic Radiology
|March 19, 2010
Summary
Comparing performance metrics for mammography interpretation, this study found that binary (yes/no) ratings offered larger differences in performance comparisons than receiver-operating characteristic (ROC)-type ratings, especially when considering the observer
Area of Science:
- Radiology and Medical Imaging
- Observer Performance Studies
- Statistical Analysis in Medical Research
Background:
- Evaluating diagnostic performance is crucial for medical imaging, particularly for breast cancer screening technologies like full-field digital mammography (FFDM) and digital breast tomosynthesis (DBT).
- Observer performance studies rely on rating data to compare imaging modalities, with different data types (binary vs. ROC-type) potentially influencing results.
Purpose of the Study:
- To compare performance assessment methods using binary and ROC-type rating data in observer studies.
- To evaluate differences in comparing full-field digital mammography (FFDM) versus FFDM plus digital breast tomosynthesis (DBT).
- To understand how rating scale choice impacts the detection of performance differences between imaging techniques.
Main Methods:
- Eight radiologists interpreted 77 digital mammographic examinations using both binary (yes/no) and ROC-type (0-100) rating scales.
- Performance was quantified using the area under the ROC curve (AUC) for ROC-type data and Youden's index for binary data.
- Differences in reader-averaged AUCs between FFDM and FFDM+DBT were compared, considering reader variability.
Main Results:
- Absolute differences in performance metrics were larger on average for binary ratings (0.12) compared to ROC-type ratings (0.07).
- Standardized differences also indicated larger effects for binary ratings (2.32 vs. 1.63).
- The observed discrepancies were influenced by the specific operating points chosen by individual readers.
Conclusions:
- The choice of an observer's operating point is critical when designing performance studies.
- While ROC-type ratings offer detailed information, binary ratings may better reflect actual observer behavior and provide statistical advantages in certain applications.
- Binary response analysis can offer statistical advantages in specific clinical scenarios for observer performance studies.
Related Concept Videos
01:15Receiver Operating Characteristic Plot
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
01:21Dose Response Curve: Conventional Versus Nonmonotonic
The correlation between a drug's dosage and its impact on a biological system is a cornerstone of pharmacology and toxicology. Conventional dose–response curves, which include graded and quantal relationships, are key to this understanding. Graded dose–response curves depict the spectrum of a biological reaction to different doses within an individual, indicating that as the drug dosage increases, so does the intensity of the response. On the other hand, quantal dose–response relationships...
01:23Actor-Observer Effect
The actor-observer effect, a cognitive bias closely linked to the fundamental attribution error, refers to the tendency for individuals to attribute their behavior to external, situational factors while explaining others’ behavior in terms of internal, dispositional traits. This asymmetry in attribution significantly influences social perception and judgment.Cognitive Mechanisms Behind the EffectTwo primary psychological mechanisms contribute to the actor-observer effect: differences in visual...
01:20Region of Convergence of Laplace Tarnsform
The Region of Convergence (ROC) is a fundamental concept in signal processing and system analysis, particularly associated with the Laplace transform. The ROC represents an area in the complex plane where the Laplace transform of a given signal converges, determining the transform's applicability and utility.
Consider a decaying exponential signal that begins at a specific time. When deriving its Laplace transform, the time-domain variable is replaced with a complex variable. This substitution...
Consider a decaying exponential signal that begins at a specific time. When deriving its Laplace transform, the time-domain variable is replaced with a complex variable. This substitution...
01:11Observational Studies
Observational studies are a type of analytical study where researchers observe events without any interventions. In other words, the researcher does not influence the response variable or the experiment's outcome.
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One example of...
There are three types of observational studies – Prospective, retrospective, and cross-sectional.
Prospective Study
Prospective studies, also known as longitudinal or cohort studies, are carried out by collecting future data from groups sharing similar characteristics. One example of...
01:03Dose-Response Relationship: Overview
Agonists can bind with and activate receptors, resulting in the formation of drug-receptor complexes. Once formed, these complexes catalyze many biochemical processes at the cellular level and subsequently induce a pharmacologic response. The degree of response is directly proportional to the fraction of activated receptors, which in turn, depends on the concentration of the drug at the receptor site as well as the sensitivity of the receptor. An increase in the administered dose contributes to...