Graph enhanced ContextFusion-EmoNet (CFEN): integrating facial, postural, and environmental cues for emotion...

Emotion recognition in videos is still difficult due to the fact that affective cues are spread throughout facial expressions, body movements and scene context. Face only approaches tend to perform poorly in most cases because of being affected by such factors as varying illumination, occlusion, camera motion, dynamic backgrounds, and multiple interacting subjects. To overcome these drawbacks, a novel and flexible model called ContextFusion-EmoNet (CFEN) is proposed in this paper to utilize appearance, motion and relational information for robust emotion recognition from video. CFEN uses an Average Contextual Loss (ACL) guided key-frame selection method, incorporating the differences of the VGG16 features and Farneback optical flow weight to select 4 key frames that are rich in motion and extremely informative per video. The selected images are then fed into ResNet18 feature extractor, a temporal context modeler, which is a Transformer encoder, and a deterministic Graph Convolutional Network (GCN) module for modeling the relation between frame tokens. Good intra-dataset performance is shown through extensive experimentation on the four datasets, CAER, CK + , DFEW, and FERV39k, with accuracy rates of 97.71%, 98.00%, 91.43% and 89.71% respectively.