Enhancing action quality assessment via multi-level action difference learning

Action Quality Assessment (AQA) is a crucial task in fine-grained video understanding with wide applications in domains like sports analytics, skill training, and rehabilitation. The core challenge lies in discerning subtle action differences in videos with low intra-class discrimination. While recent methods have made progress, they primarily rely on globally-pooled features to model fine-grained semantics. This approach creates a fundamental bottleneck, as global representations often average out the critical local temporal variations and spatial details that differentiate action quality. To address this issue, we propose a novel Multi-level Action Difference Learning (MADL) framework. Instead of distilling information from a single global view, MADL systematically performs coarse-to-fine action difference contrasts at global, temporal, and spatial levels, offering comprehensive assessments. Moreover, we propose a Temporal Alignment Video Encoder (TAVE) that integrates context-related temporal dynamic adaptation, facilitating temporal matching for action contrasts. To enhance the discriminative power of the multi-level representations, a Global-guided Feature Refinement Module (GFRM) is then designed to optimize fine-grained representations guided by global features. Extensive experiments on five public benchmarks (FineDiving, MTL-AQA, JIGSAWS, Rhythmic Gymnastics, and Fis-V) demonstrate that our method achieves state-of-the-art performance across both short-term and long-term assessment tasks. The source code will be publicly available at https://github.com/XuHuangbiao/MADL.