Time to retire F1-binary score for action unit detection.

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Action unit detection is crucial for facial expression recognition, as expressions can be broken down into individual action units.
• The F1-binary score is commonly used to evaluate action unit detection models.
• Class imbalance poses a significant challenge in machine learning tasks, including face analysis.

Purpose of the Study:

• To argue against the use of the F1-binary score for evaluating action unit detection models.
• To demonstrate the negative impact of class imbalance on the reliability of the F1-binary score.
• To propose and justify the F1-micro score as a more suitable replacement metric.

Main Methods:

• Investigated the influence of class imbalance on action unit detection performance.
• Evaluated the impact of class imbalance in training sets, testing sets, and on generalizability to new data.
• Conducted empirical analyses to compare the performance of F1-binary and F1-micro scores under imbalanced conditions.

Main Results:

• Class imbalance significantly undermines the reliability of the F1-binary score in action unit detection.
• The F1-binary score provides a misleading evaluation when dealing with imbalanced datasets common in face analysis.
• Empirical evidence supports the superiority of the F1-micro score in accurately reflecting model performance.

Conclusions:

• The F1-binary score should be retired as an evaluation metric for action unit detection due to its susceptibility to class imbalance.
• The F1-micro score is a more robust and reliable metric for evaluating action unit detection models, especially in the presence of class imbalance.
• Adopting F1-micro will lead to more accurate assessments and advancements in facial expression recognition systems.