Explainable Action Form Assessment by Exploiting Multimodal Chain-of-Thoughts Reasoning.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Current video understanding methods fail to assess action standardization and provide actionable feedback.
• Existing datasets lack labels for action standardization degree and explainable feedback for quality assessment.

Purpose of the Study:

• Define a new Human Action Form Assessment (AFA) task.
• Introduce the CoT-AFA dataset with multi-level annotations for fitness and martial arts videos.
• Develop an Explainable Fitness Assessor (EFA) framework for action judgment, explanation, and solution provision.

Main Methods:

• Introduced a novel Chain-of-Thought explanation paradigm for comprehensive reasoning.
• Proposed the Explainable Fitness Assessor (EFA) framework with parallel processing streams and a dynamic gating mechanism.
• Fused visual and semantic information for enhanced analytical capabilities.

Main Results:

• Achieved significant improvements in explanation generation (+16.0% in CIDEr).
• Enhanced action classification accuracy by +2.7%.
• Improved action quality assessment accuracy by +2.1%.

Conclusions:

• The CoT-AFA dataset and EFA framework show great potential for advancing human action analysis and feedback.
• The proposed method offers a comprehensive approach to action assessment, explanation, and improvement.