Measuring interobserver variation in a pathology EQA scheme using weighted κ for multiple readers.

Area of Science:

• Uropathology
• Biostatistics
• Quality Assurance

Background:

• Urological Pathology External Quality Assurance (EQA) schemes identify observer variation in prostate biopsy diagnosis and grading.
• Traditional kappa (κ) statistics treat all disagreements equally, potentially overestimating variation.

Purpose of the Study:

• To implement customized weighting schemes for kappa statistics in a prostate EQA scheme.
• To better reflect interobserver agreement by accounting for the degree of discordance.

Main Methods:

• Three web-based circulations involved 83-116 pathologists, categorized as expert or other readers.
• Analyses included 10-8 diagnosis cases and 5-6 Gleason Sum Score cases (invasive only).
• Pairwise-weighted kappa (κ(p-w)) was used for multiple readers with customized weighting.

Main Results:

• Composite kappa and pairwise-weighted kappa for diagnosis indicated 'almost perfect' agreement (0.86 and 0.91, respectively).
• Gleason Sum Score analysis showed lower agreement (κ=0.38, κ(p-w)=0.58), with discrepancies often between adjacent grades.
• Weighted kappa highlighted partial agreement, suggesting less clinical impact of some grading differences.

Conclusions:

• Weighted kappa statistics provide a more nuanced measure of agreement in uropathology EQA.
• Customized weighting improves the assessment of interobserver agreement by considering discordance severity.
• While diagnostic agreement is high, grading agreement remains a challenge, though weighted analysis suggests less clinical significance for minor discrepancies.