Facial obstructions and baseline correction shape affective computing's detection of emotion-behavior relationships

Affective computing (AC) is increasingly used to study emotional processes underlying decision-making, yet its methodological validity in capturing spontaneous emotional responses and their behavioral relevance remains debated. In particular, it is unclear how facial obstructions and baseline correction affect the accuracy of AC-based facial emotion measures in capturing emotion-behavior relationships.

The predictive validity of AC-based facial emotion measures was evaluated in charitable decision-making. Participants (N = 88) viewed dog images and made voluntary donations. Facial expressions were recorded using FaceReader (FR); in Group 1 (n = 43), facial electromyography (EMG) electrodes were also applied, while Group 2 (n = 45) had no electrodes. FR results were compared with EMG and self-report (SR) measures, and the effects of facial obstructions (presence of EMG electrodes) and baseline correction on AC accuracy were examined.

Donation behavior and SR emotion ratings were equivalent across both groups, whereas FR measurements differed. Corrugator EMG activity negatively correlated with SR and FR valence in both groups, but associations with FR were stronger when facial electrodes were absent, indicating that facial obstructions reduce AC accuracy. Across methods, stronger negative emotions were associated with higher donation amounts. FR valence and happiness correlated with donations across both samples, with stronger associations in the group without EMG electrodes; links with sadness and anger emerged only when facial expressions were unobstructed. Accounting for the prestimulus baseline improved convergence between FR and self-reported emotions and strengthened associations between FR-based valence and anger with donation behavior.

Results indicate that AC can capture prominent emotion-behavior relationships comparable to EMG and SR when recording conditions are optimal, but accuracy is reduced by facial obstructions and enhanced by baseline correction. These methodological factors are critical to consider in multimodal studies and research linking emotions to behavior.