Related Experiment Videos
Agreement between experts: an answer, but is it the answer for you?
1Department of Psychology, Macquarie University, Sydney, Australia. phutchin@bunyip.bhs.mq.edu.au
Summary
Measuring expert agreement on ordinal scales requires careful consideration. Polychoric correlation offers a method, but defining agreement remains subjective, especially with ordinal data.
Area of Science:
- Statistics
- Bibliometrics
- Scientific Publishing
Background:
- Assessing inter-rater reliability for subjective evaluations is crucial in scientific publishing.
- Ordinal scales are frequently used by experts to rate manuscript quality, posing unique measurement challenges.
Purpose of the Study:
- To discuss methods for measuring agreement between experts using ordinal scales.
- To highlight the utility of polychoric correlation for ordinal data.
- To emphasize the importance of carefully defining agreement metrics in research.
Main Methods:
- Discussion of statistical concepts for measuring agreement.
- Application of polychoric correlation as a potential method.
- Illustrative example using article quality ratings for the Journal of Clinical Neuroscience.
Main Results:
- Polychoric correlation is identified as a valuable statistical tool for ordinal data.
- Availability of computer programs facilitates the estimation of polychoric correlation.
- The study underscores that the definition of agreement is context-dependent and a matter of careful choice.
Conclusions:
- While polychoric correlation is useful, it does not negate the need for thoughtful definition of agreement.
- Ordinal categories present inherent difficulties that must be acknowledged in statistical analysis.
- Researchers must critically evaluate measurement properties, even with advanced statistical techniques.