A computer program for assessing interexaminer agreement when multiple ratings are made on a single subject

Area of Science:

• Statistics
• Medical Informatics

Background:

• Assessing inter-rater reliability is crucial for the validity of clinical evaluations.
• Existing methods may not fully capture nuanced agreement levels among multiple evaluators.

Purpose of the Study:

• To present a novel statistical method and accompanying computer program for quantifying inter-rater agreement.
• To provide a comprehensive tool for evaluating reliability in multi-examiner scenarios.

Main Methods:

• Development of a computer program implementing a new statistical method.
• Calculation of overall agreement percentage considering partial agreements.
• Pairwise examiner agreement assessment.
• Analysis of individual examiner deviation from the average rating using Z statistics.

Main Results:

• The method provides an overall agreement percentage accounting for partial agreement.
• Individual examiner agreement with peers is quantified.
• Deviations from the group average are assessed using Z statistics.
• Statistical and clinical criteria for evaluating agreement levels are proposed.

Conclusions:

• The developed statistical method and program offer a robust approach to inter-rater reliability assessment.
• This tool aids in evaluating the consistency and quality of multi-examiner evaluations.
• The criteria provided facilitate the interpretation of agreement levels in research and clinical practice.