Enhancing classroom behavior analysis with multimodal data: a cross-attention fusion network approach

Area of Science:

• Artificial Intelligence
• Educational Technology
• Computer Science

Background:

• Classroom behavior analysis is vital for understanding student engagement and instructional quality in higher education.
• Existing methods face challenges with complex settings, large datasets, and single-modality limitations.

Purpose of the Study:

• To develop a multimodal deep learning framework for accurate student classroom behavior classification.
• To address limitations of current approaches by integrating diverse data sources.

Main Methods:

• Developed a Multi-Level Cross-Self Attention Fusion Network (MCSFN) integrating visual and physiological data.
• Utilized three branches for spatial visual features, Restormer-enhanced video, and physiological signals.
• Employed multi-level cross-attention for robust feature fusion.

Main Results:

• Achieved 95.88% classification accuracy on a self-constructed dataset.
• Outperformed single-modality baseline methods significantly.
• Demonstrated the effectiveness of multimodal fusion for behavior recognition.

Conclusions:

• The MCSFN framework offers a practical pathway for data-informed and adaptive classroom practices.
• Highlights the potential of AI-driven analytics for instructional design and personalized learning.
• Enables real-time monitoring and enhanced personalized learning interventions.