Visual language transformer framework for multimodal dance performance evaluation and progression monitoring

Area of Science:

• Computer Science
• Artificial Intelligence
• Robotics
• Human-Computer Interaction

Background:

• Dance performance evaluation is complex, requiring coordination of movement and musicality.
• Existing automatic assessment methods often focus on single tasks and simple movements.
• There's a need for advanced models to handle multi-modal, multi-task dance analysis.

Purpose of the Study:

• To develop a novel transformer-based visual-language framework for multi-modal dance performance evaluation.
• To address challenges in learning representations for complex, synchronized dance movements across diverse styles and expertise levels.
• To capture the multi-task nature of dance assessment, including classification, quality estimation, and music synchronization.

Main Methods:

• Integration of contrastive self-supervised learning, spatiotemporal graph convolutional networks (STGCN), and long short-term memory networks (LSTM).
• Utilizing transformer-based text prompting for multi-task evaluations.
• Employing primitive-based segmentation and multi-modal inputs for feature extraction and analysis.

Main Results:

• Achieved 75.20 in multilabel dance classification, a 10.25% improvement over prior methods.
• Demonstrated a 92.09% lower loss in dance quality estimation compared to CotrastiveDance.
• Excelled in dance-music synchronization with a score of 2.52, outperforming CotrastiveDance by 48.67%.

Conclusions:

• The proposed framework effectively evaluates complex dance movements and music synchronization.
• The multi-task approach enhances sensorimotor skill assessment and motion analysis capabilities.
• This research advances automatic dance performance assessment through innovative AI techniques.