Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Assessing rater performance without a "gold standard" using consensus theory

S C Weller1, N C Mann

  • 1Department of Preventive Medicine and Community Health, University of Texas Medical Branch, Galveston 77555-1153, USA.

Medical Decision Making : an International Journal of the Society for Medical Decision Making
|January 1, 1997
PubMed
Summary
This summary is machine-generated.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Factors affecting Agrobacterium tumefaciens-mediated transformation of peppermint.

Plant cell reports·2019
Same author

Transgenic peppermint (Mentha×piperita L.) plants obtained by cocultivation with Agrobacterium tumefaciens.

Plant cell reports·2019
Same author

Stability and expression of amplified EPSPS genes in glyphosate resistant tobacco cells and plantlets.

Plant cell reports·2013
Same author

Rehabilitation for traumatic brain injury in children and adolescents.

Evidence report/technology assessment (Summary)·2004
Same author

Hospital follow-up of patients categorized as not needing an ambulance using a set of emergency medical technician protocols.

Prehospital emergency care·2001
Same author

A population-based assessment of pediatric all-terrain vehicle injuries.

Pediatrics·2001
Same journal

Flexible Survival Extrapolation with Blended Hazards: Accounting for Treatment Effect Waning in Health Technology Assessment.

Medical decision making : an international journal of the Society for Medical Decision Making·2026
Same journal

A Microsimulation Model for Chronic Kidney Disease Progression in Type 2 Diabetes Patients in the United States: Michigan Model for Diabetes-Chronic Kidney Disease Model.

Medical decision making : an international journal of the Society for Medical Decision Making·2026
Same journal

Cardiovascular Risk Estimation and Statin Adherence: A Historical Cohort Study.

Medical decision making : an international journal of the Society for Medical Decision Making·2026
Same journal

Taste or Scale? Methodological Approach to Health Preferences Comparison across Groups.

Medical decision making : an international journal of the Society for Medical Decision Making·2026
Same journal

Mind the Gap: Impact of New Labels on Public Perceptions and Calculated Risk of Adverse Outcomes after a Melanoma In Situ Diagnosis-A Secondary Analysis of an Online Randomized Experiment.

Medical decision making : an international journal of the Society for Medical Decision Making·2026
Same journal

A Metamodel-Based General-Purpose Autocalibration Tool for Simulation Models.

Medical decision making : an international journal of the Society for Medical Decision Making·2026
See all related articles

Consensus theory effectively assesses rater diagnostic performance and estimates diagnoses without a gold standard. This method remains robust even with biased raters, as shown in a study of elbow radiographs.

Area of Science:

  • Medical imaging analysis
  • Diagnostic accuracy assessment
  • Statistical modeling in medicine

Background:

  • Evaluating diagnostic performance often relies on a gold standard, which is not always available.
  • Rater variability and bias can significantly impact diagnostic accuracy.
  • Consensus theory offers a framework for pooling rater information.

Purpose of the Study:

  • To demonstrate the application of consensus theory for assessing rater diagnostic performance.
  • To estimate case diagnoses when a definitive criterion standard is absent.
  • To evaluate the robustness of consensus models under biased rater conditions.

Main Methods:

  • Utilized consensus theory to pool and weight information from multiple raters.
  • Employed Monte Carlo simulations with 1,200 datasets to test model robustness.

Related Experiment Videos

  • Applied the consensus model to a dataset of elbow radiographs for illustration.

    • Main Results:

    • Consensus theory successfully estimated rater competencies and case diagnoses.
    • The model demonstrated robustness, providing accurate estimates even with biased rater responses.
    • Comparison with follow-up data validated the consensus-model estimates.

    Conclusions:

    • Consensus theory is a valuable tool for diagnostic performance assessment in the absence of a gold standard.
    • The model's ability to handle biased raters enhances its practical applicability in medical diagnostics.
    • Accurate case diagnoses and rater competencies can be reliably estimated using this approach.