Text-to-Motion Generation With Discrete Representations and Large Language Models

Based on Vector Quantized Variational AutoEncoder (VQ-VAE) and Transformers, we investigate a simple yet effective conditional generative framework for text-to-motion generation. First, we find that, with a few training recipes (EMA and Code Reset), a CNN-based VQ-VAE can learn high-quality discrete motion representations, which largely facilitate motion generation. Second, to better understand motion descriptions, we propose T2M-GPT+ that uses Large Language Models (LLMs) to extract text features. Further, we propose T2M-GIT+, which employs a non-autoregressive method to generate discrete motion representations in parallel, and is therefore more efficient than T2M-GPT+ while achieving comparable results. Comprehensive experiments show that our method is superior to existing methods, including competitive diffusion methods. For example, we achieve a Top-1 R-precision of 0.53 and an FID of 0.10 on the HumanML3D dataset, substantially outperforming MotionDiffuse, which obtains a Top-1 R-Precision of 0.49 and an FID of 0.63. Additionally, we conduct analyses and find that, even for the largest dataset, i.e., HumanML3D, it may still limit the performance of the proposed approach. This suggests that a larger dataset could bring additional improvement to our approach.