Language as a Universal Anchor for Cross-Domain Sketch-Based 3D Human Mesh Recovery
Related Experiment Videos
Abstract:
When humans sketch the same pose, no two drawings are alike. Synthetic sketches exhibit algorithmic precision with clean edges and consistent strokes, while freehand sketches diverge wildly-each bearing the unique abstraction, style, and imperfections of its creator. This fundamental divergence has long challenged 3D human reconstruction systems, which struggle to bridge the chasm between these disparate visual domains. We present a paradigm shift: while sketches diverge, language converges. A pose described as "arms raised overhead" carries the same semantic meaning whether drawn by algorithm or artist. Building on this insight, we introduce a universal feature anchor-natural language-that remains constant across visual variations. Our framework leverages text descriptions to guide feature learning, creating cross-domain representations that transcend the synthetic-freehand divide. At the technical core lies our Text-based Body Pose Head (TBPH), featuring a novel gating mechanism where language-derived features dynamically reweight spatial regions of sketch features. This textguided attention enables the model to focus on semantically meaningful pose indicators while suppressing domain-specific noise and stylistic artifacts. By augmenting 26,000 sketch-pose pairs with rich textual descriptions, we enable cross-modal supervision that teaches our model to see past surface differences to underlying pose semantics. Extensive experiments demonstrate our method's superiority: with ideal descriptions, UniAnchor achieves 139.86 mm MPJPE on freehand sketches, a 4.5% improvement over TokenHMR, and further outperforms it by 11.08% in zero-shot generalization. With casual descriptions, our framework still retains a substantial portion of these gains, indicating that much of the language benefit is effectively internalized through multimodal training. More importantly, our model remains robust on highly abstract amateur sketches, demonstrating the practical value of multimodal supervision for cross-domain 3D human reconstruction.
Related Concept Videos
Modeling and Similitude
Mesh Analysis
A fundamental concept in mesh analysis is the definition of meshes and mesh currents. A mesh is a closed...
Divergence Theorem in 3D Space