On the Dynamics of Classification Measures for Imbalanced and Streaming Data

Area of Science:

• Machine Learning
• Data Science
• Computer Science

Background:

• Classifier evaluation metrics must be chosen carefully for imbalanced datasets.
• Existing research often overlooks how varying imbalance ratios affect metric performance.
• Imbalanced data streams present unique challenges not fully explored by current metrics.

Purpose of the Study:

• To analyze how different class proportions influence the behavior and dynamics of classifier evaluation metrics.
• To investigate the impact of changing class proportions on metrics in imbalanced data streams.
• To develop a unified interpretation for various metrics across different class distributions.

Main Methods:

• Visualizing measure probability mass functions and gradients across diverging class proportions.
• Analyzing the dynamics of eight popular classification measures under varying imbalance ratios.
• Proposing a novel histogram-based normalization method for unified metric interpretation.

Main Results:

• Each classification measure exhibits distinct changes in behavior with varying class proportions.
• Class proportions significantly affect measure values, distributions, and gradients.
• A direct link between class ratio shifts and specific concept drift types was identified.

Conclusions:

• Classifier evaluation in imbalanced scenarios necessitates considering the specific impact of class proportions on chosen metrics.
• The findings provide insights into measure dynamics, crucial for developing robust classifiers and drift detectors for imbalanced data streams.
• The proposed normalization method offers a standardized way to interpret metrics across diverse imbalanced datasets.