Facing Imbalanced Data Recommendations for the Use of Performance Metrics

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Facial action unit (AU) recognition is crucial for automated video analysis.
• Existing research primarily focuses on face tracking, registration, and feature selection.
• The impact of imbalanced data on AU detection performance metrics remains underexplored.

Purpose of the Study:

• To investigate the influence of data skew on performance metrics for facial action unit detection.
• To evaluate how skewed data distributions affect various threshold and rank metrics.
• To propose methods for mitigating bias in performance evaluations.

Main Methods:

• Experiments were conducted using simulated classifiers and three diverse facial action unit databases.
• The study analyzed the effect of data skew on threshold metrics (Accuracy, F-score, Cohen's kappa, Krippendorf's alpha).
• Rank metrics, including area under the ROC curve and precision-recall curve, were also evaluated.

Main Results:

• Skewed data distributions significantly attenuated most evaluated metrics, except for the area under the ROC curve.
• Precision-recall curves indicated that ROC might obscure poor performance in skewed datasets.
• The degree of performance attenuation varied across different metrics and datasets.

Conclusions:

• Data skew is a critical factor that can bias performance evaluations in facial action unit detection.
• Standard performance metrics can be misleading when dealing with imbalanced datasets.
• Reporting skew-normalized scores is recommended to obtain more reliable performance estimates.