Appropriate Statistics for Determining Chance-Removed Interpractitioner Agreement.

Area of Science:

• Statistics
• Medical Diagnostics
• Traditional Chinese Medicine

Background:

• Fleiss' Kappa (FK) is frequently misapplied as the standard for chance-removed inter-rater agreement with ordinal data.
• This incorrect application can lead to misleading conclusions in inter-rater agreement research.
• A novel Chinese Medical (CM) diagnostic format, the Diagnostic System of Oriental Medicine (DSOM), was developed to address limitations in current diagnostic recording.

Purpose of the Study:

• To demonstrate the inappropriate results produced by Fleiss' Kappa (FK) under specific conditions.
• To propose Gwet's AC2 as a more appropriate statistic for evaluating inter-rater agreement, particularly with ordinal data.
• To compare the performance of FK and Gwet's AC2 using a novel CM diagnostic system (DSOM) that accounts for chance agreement.

Main Methods:

• Five CM practitioners diagnosed 42 subjects using the DSOM format.
• Inter-rater agreement was evaluated using linearly weighted simple agreement, Fleiss' Kappa (FK), and Gwet's AC2.
• Data were analyzed overall and then sorted into groups to demonstrate the effect of data marginality on agreement statistics.

Main Results:

• Overall agreement was consistently lower when calculated using FK compared to Gwet's AC2.
• The discrepancy between FK and AC2 increased with more 'free marginal' data.
• FK results are unreliable unless data are fixed marginal; AC2 is appropriate for both fixed and free marginal data.

Conclusions:

• Fleiss' Kappa (FK) is an inappropriate statistic for evaluating inter-rater agreement with ordinal data when marginality is not fixed.
• Gwet's AC2 provides a more accurate and reliable measure of inter-rater agreement in such scenarios.
• Gwet's AC2 should be adopted as the standard statistical approach for determining agreement between practitioners, especially in CM diagnostics.