A hierarchical transformer-graph framework for explainable student engagement estimation in E-learning videos

The importance of understanding student engagement in learning with AI technologies is particularly relevant for the case of adaptive learning. Most current vision-based methods rely solely on appearance-based emotion detection using facial expressions. They do not consider the geometric and behavioral temporal dynamics that can occur in unconstrained webcam streaming. The main contribution of this paper is the formulation of a hierarchical method based on Transfomer-Graphs to perform engagement-aware affective inference. The dual architecture proposed here consists of a BEiT-based vision transformer and a graph attention network (GAT). The vision transformer is in charge of capturing the global facial features. The GAT is in charge of modeling the relational dependencies of the 468 facial landmarks. The affectNet model is trained to learn robust facial emotions with high representation capabilities. The emotions predicted in the model do not directly supervise affective engagement. Instead, via a rule-based method of affective model fusion the predicted emotions are engaged with relevant affective states. When combined with gaze-based attention and eye aspect ratio measures that reflect alertness, these predictors infer frame-level engagement. Temporal smothing is applied to these predictors to provide a video-level trajectory. Experimental results show that the proposed Transformer-Graph fusion architecture achieves robust facial emotion recognition with 88% accuracy and interpretable decision-making behaviour in real-world webcam settings. The method is suitable for webcam video. Most of the current methods rely on behavioral, or geometric features, whereas this method paramaterizes the matrix as a behavioral features overlay. This method does need to clarified as describe for the engagement dataset.