Related Experiment Video
Updated: May 10, 2025
10:33
Expedited Radiation Biodosimetry by Automated Dicentric Chromosome Identification ADCI and Dose Estimation
Published on: September 4, 2017
15.6K
Drivers of bias in diagnostic test accuracy estimates when using expert panels as a reference standard: a simulation
B E Kellerhuis1, K Jenniskens2, E Schuit2
1Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, The Netherlands. b.e.kellerhuis@umcutrecht.nl.
BMC Medical Research Methodology
|April 24, 2025
Summary
Expert panels are crucial for diagnostic test accuracy research when no gold standard exists. Simulation results show that less accurate component reference tests significantly increase bias in accuracy estimates.
Area of Science:
- Medical Diagnostics
- Biostatistics
- Health Research Methodology
Background:
- Expert panels serve as a reference standard in diagnostic test accuracy research, particularly when a definitive gold standard is absent.
- Key study and expert panel characteristics influencing diagnostic accuracy estimates remain incompletely understood.
- This study addresses the need to clarify factors affecting the reliability of diagnostic accuracy estimates derived from expert panels.
Purpose of the Study:
- To assess the impact of various study and expert panel characteristics on the accuracy estimates of index diagnostic tests.
- To quantify the bias introduced by different simulation scenarios in diagnostic test accuracy research.
- To identify factors that contribute to reliable or unreliable accuracy estimates when using expert panels as a reference.
Main Methods:
- Simulations were conducted using an expert panel as the reference standard to evaluate an index diagnostic test's sensitivity and specificity.
- Diagnostic accuracy was determined by aggregating expert probability estimates from four component reference tests.
- Scenarios varied target condition prevalence, component test accuracy, panel size, study population size, and inter-expert variability.
Main Results:
- Bias in accuracy estimates was minimally affected by study population size or expert panel size.
- Target condition prevalence significantly impacted bias; lower prevalence (0.2) resulted in wider, often lower, sensitivity and specificity estimates compared to higher prevalence (0.5).
- Improved accuracy of component reference tests (80% vs. 70% sensitivity/specificity) demonstrably reduced bias in index test accuracy estimates.
Conclusions:
- The accuracy of component reference tests is a critical determinant of bias in expert panel-based diagnostic accuracy estimates; less accurate tests increase bias.
- While prevalence and inter-expert variability also influence bias, their impact can vary in magnitude and direction across different scenarios.
- Understanding these factors is essential for improving the reliability of diagnostic test accuracy research that relies on expert consensus.
Related Concept Videos
01:29Bias in Epidemiological Studies
92
Biases can arise at various stages of research, from study design and data collection to analysis and interpretation. Recognizing and addressing these biases is essential to ensure the validity and reliability of epidemiological findings.Broadly speaking, biases in epidemiology fall into three main categories: selection bias, information bias, and confounding. A more detailed description of possible biases is:
92
01:22Bias
3.7K
Bias refers to any tendency that prevents a question from being considered unprejudiced. In research, bias occurs when one outcome or answer is selected or encouraged over others in sampling or testing. Bias can occur during any research phase, including study design, data collection, analysis, and publication.
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
In statistics, a sampling bias is created when a sample is collected from a population, and some members of the population are not as likely to be chosen as others (remember, each member...
3.7K
01:13Accuracy and Errors in Hypothesis Testing
153
Hypothesis testing is a fundamental statistical tool that begins with the assumption that the null hypothesis H0 is true. During this process, two types of errors can occur: Type I and Type II. A Type I error refers to the incorrect rejection of a true null hypothesis, while a Type II error involves the failure to reject a false null hypothesis.
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
In hypothesis testing, the probability of making a Type I error, denoted as α, is commonly set at 0.05. This significance level indicates a 5%...
153
01:13Sensitivity, Specificity, and Predicted Value
135
In healthcare diagnostics, laboratory tests play a crucial role in identifying and diagnosing a wide range of medical conditions. However, interpreting test results is not always straightforward. An abnormal test result does not always confirm the presence of a disease, just as a normal result does not guarantee its absence. To assess the reliability of these diagnostic tools, healthcare practitioners rely on two key statistical indicators: sensitivity and specificity.
Sensitivity is the...
Sensitivity is the...
135
01:15Receiver Operating Characteristic Plot
64
A ROC (Receiver Operating Characteristic) plot is a graphical tool used to assess the performance of a binary classification model by illustrating the trade-off between sensitivity (true positive rate) and specificity (false positive rate). By plotting sensitivity against 1 - specificity across various threshold settings, the ROC curve shows how well the model distinguishes between classes, with a curve closer to the top-left corner indicating a more accurate model. The area under the ROC curve...
64
01:19Testing a Claim about Standard Deviation
2.4K
A complete procedure to test a claim about population standard deviation or population variance is explained here.
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
The hypothesis testing for the claim of population standard deviation (or variance) requires the data and samples to be random and unbiased. The population distribution also must be normal. There is no specific requirement on the sample size as the estimation is based on the chi-square distribution.
As a first step, the hypothesis (null and alternative) concerning the claim about...
2.4K