Jove
Visualize
Contact Us
JoVE
x logofacebook logolinkedin logoyoutube logo
ABOUT JoVE
OverviewLeadershipBlogJoVE Help Center
AUTHORS
Publishing ProcessEditorial BoardScope & PoliciesPeer ReviewFAQSubmit
LIBRARIANS
TestimonialsSubscriptionsAccessResourcesLibrary Advisory BoardFAQ
RESEARCH
JoVE JournalMethods CollectionsJoVE Encyclopedia of ExperimentsArchive
EDUCATION
JoVE CoreJoVE BusinessJoVE Science EducationJoVE Lab ManualFaculty Resource CenterFaculty Site
Terms & Conditions of Use
Privacy Policy
Policies

Related Experiment Videos

Detecting systematic bias between two raters.

John Ludbrook1

  • 1The University of Melbourne, Parkville, 563 Canning Street, Carlton North, Victoria 3054, Australia. ludbrook@bigpond.net.au

Clinical and Experimental Pharmacology & Physiology
|February 6, 2004
PubMed
Summary

This study introduces a new method to evaluate the bias index (BI) for comparing two raters on ordered categorical scales. The bias index accounts for agreement extent, addressing limitations of previous bias detection methods.

Related Concept Videos

You might also read

Related Articles

Articles linked to this work by shared authors, journal, and citation graph.

Sort by
Same author

Vascular geometry and oxygen diffusion in the vicinity of artery-vein pairs in the kidney.

American journal of physiology. Renal physiology·2014
Same author

Letter to the editor. Second thoughts on the sidedness of P.

Clinical and experimental pharmacology & physiology·2013
Same author

Telemetry-based oxygen sensor for continuous monitoring of kidney oxygenation in conscious rats.

American journal of physiology. Renal physiology·2013
Same author

Should we use one-sided or two-sided P values in tests of significance?

Clinical and experimental pharmacology & physiology·2013
Same author

Analysing 2 × 2 contingency tables: which test is best?

Clinical and experimental pharmacology & physiology·2013
Same author

A primer for biomedical scientists on how to execute model II linear regression analysis.

Clinical and experimental pharmacology & physiology·2011

Area of Science:

  • Statistics
  • Biostatistics
  • Psychometrics

Background:

  • The kappa statistic is commonly used for inter-rater reliability on ordered categorical scales.
  • A key limitation of the kappa statistic is its inability to detect systematic bias between raters.
  • Existing methods for bias detection, such as McNemar's test and marginal homogeneity tests, have limitations.

Purpose of the Study:

  • To address the limitations of existing methods for detecting inter-rater bias.
  • To propose a satisfactory method for evaluating the bias index (BI), which considers both bias and agreement.
  • To provide a statistically sound approach for assessing rater agreement and bias.

Main Methods:

  • Review of existing methods for comparing two raters on ordered categorical scales.

Related Experiment Videos

  • Critique of the kappa statistic and other bias detection tests (modified McNemar, single binomial, symmetry of disagreement index, marginal homogeneity test).
  • Development and description of a novel method for evaluating the bias index (BI).

    • Main Results:

    • The kappa statistic and previously suggested bias detection methods fail to adequately address inter-rater bias.
    • These methods often ignore the extent of agreement between raters.
    • A new method is presented for the evaluation of the bias index (BI), incorporating agreement extent.

    Conclusions:

    • Existing methods for assessing inter-rater reliability and bias are insufficient.
    • The proposed method for evaluating the bias index (BI) offers a more comprehensive approach.
    • This new method enhances the analysis of rater agreement and systematic bias.