Controllable Music-to-Dance Generation with Hierarchical Motion Representation Learning

Current music-to-dance generation methods mainly rely on musical features, limiting precise control over generated movements. In particular, most existing methods with control mechanisms do not support example-based control, in which a user provides a reference motion sequence and the generated dances follow its fine-grained motion patterns while adapting to different musical pieces. While several motion-guided editing methods incorporate reference motion, they impose it as sparse positional constraints that capture local poses without modeling the reference's overall motion characteristics or adapting them to the target music. To tackle this limitation, we propose a novel framework that integrates reference motions as additional guidance for controllable dance generation while maintaining music synchronization. Our approach employs a hierarchical motion representation learning framework to capture both global characteristics and temporally coherent local details from reference sequences. A motion-music integration approach is then applied to combine the extracted motion features with a pretrained music to-dance diffusion model through efficient fine-tuning. Addition ally, cycle consistency regularization is incorporated to ensure robust generalization across diverse motion-music pairs. Extensive experiments demonstrate that our framework achieves effective controllability while maintaining high-quality motion generation comparable to state-of-the-art music-to-dance methods. The code is available at https://anonymous.4open.science/r/Controllable Music-to-Dance-FF76/.