Why Cohen's Kappa should be avoided as performance measure in classification.

Area of Science:

• Machine Learning
• Data Science
• Statistical Modeling

Background:

• Cohen's Kappa and Matthews Correlation Coefficient (MCC) are key metrics for multi-class classification performance evaluation.
• Both metrics assess classifier performance but can yield different results, especially in imbalanced datasets.

Purpose of the Study:

• To investigate the relationship between Cohen's Kappa and MCC under various data imbalances.
• To identify specific conditions where Kappa exhibits anomalous behavior compared to MCC.

Main Methods:

• Comparative analysis of Cohen's Kappa and MCC across different multi-class classification scenarios.
• Experimental study focusing on confusion matrix properties, particularly the entropy of off-diagonal elements.

Main Results:

• Cohen's Kappa and MCC are generally correlated but diverge significantly in specific unbalanced situations.
• Anomalous Kappa behavior emerges when the entropy of off-diagonal confusion matrix elements decreases, indicating a potential pitfall.

Conclusions:

• Cohen's Kappa may not be a reliable metric for comparing classifiers due to its potential for counterintuitive results.
• The findings suggest limitations in using Kappa for performance evaluation, especially with imbalanced datasets.