A scalable multimodal framework for learning engagement recognition using three-dimensional convolutional neural...

Area of Science:

• Artificial Intelligence
• Educational Technology
• Computer Vision

Background:

• Accurate assessment of student learning engagement is crucial for effective pedagogy.
• Traditional methods for engagement recognition are often subjective and labor-intensive.
• Automated systems can provide scalable and objective insights into student engagement.

Purpose of the Study:

• To develop and validate an automated system for recognizing learning engagement using behavioral and emotional cues.
• To integrate a three-dimensional convolutional neural network (3D CNN) with a semi-automatic annotation mechanism.
• To enhance the accuracy and scalability of learning engagement recognition systems.

Main Methods:

• Utilized 570 instructional video recordings, segmented into 44,059 clips.
• Employed three experts for semi-automatic annotation of emotional and behavioral engagement.
• Trained two separate 3D CNN models for emotional and behavioral feature recognition.
• Evaluated model performance using video-level validation and Fleiss' kappa for inter-annotator agreement.

Main Results:

• The emotional engagement model achieved an accuracy of 0.92.
• The behavioral engagement model achieved an accuracy of 0.80.
• Model predictions showed "almost perfect agreement" with human annotations (Fleiss' kappa).

Conclusions:

• Integrating 3D CNNs with standardized annotation rules significantly improves automated learning engagement recognition.
• The developed system demonstrates human-level reliability, accuracy, and scalability.
• The framework supports scalable AI-driven learning analytics and video-based behavior recognition.