In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis From Language Models to Physics

Area of Science:

• Computer Vision
• Artificial Intelligence
• Robotics

Background:

• Synthesizing human motion from text is crucial for digital applications.
• Current methods struggle with a trade-off between semantic accuracy and physical realism.
• Large language models (LLMs) excel at semantic understanding but lack physical realism; physics-aware models lack semantic flexibility.

Purpose of the Study:

• To develop a framework that integrates LLM-based planning with real-time physical feedback for improved motion synthesis.
• To address the limitations of existing text-to-motion generation methods by enhancing both semantic fidelity and physical plausibility.
• To enable versatile and controllable human motion synthesis from diverse textual descriptions.

Main Methods:

• Proposed In-Context Model Predictive Generation (ICMPG), a closed-loop framework.
• Utilized a Context-Aware Motion Generation (CAMG) module with an LLM for planning and candidate motion generation.
• Employed a Model Predictive Generation (MPG) module for physical simulation, semantic alignment, and reward-based selection of motion sequences.

Main Results:

• ICMPG demonstrated robust generalization across standard and zero-shot open-vocabulary settings.
• Generated motions exhibited superior physical plausibility and semantic faithfulness compared to baseline methods.
• The closed-loop refinement enabled adaptation to both semantics and physical constraints without retraining.

Conclusions:

• ICMPG effectively bridges semantic interpretation and physical simulation for text-driven motion synthesis.
• The framework offers a flexible approach adaptable to various LLM backbones.
• ICMPG represents a significant advancement in creating realistic and semantically accurate human motions from text.