Single Camera based Simulation and Try-on Ready Avatar Modeling

Reconstructing and rendering high-fidelity, animatable, and try-on ready virtual avatars using only a single camera is crucial for many applications. However, existing try-on-ready human avatar reconstruction methods struggle to achieve this from monocular videos or single images, primarily due to two issues: (1) Existing approaches typically use a single layer to represent both garments and the underlying body simultaneously, ignoring their distinct motion characteristics; (2) Input captured from a single camera often fails to fully cover both garments and the underlying body. To address these challenges, we propose a physically integrated double-layer GS-Mesh 3D clothed human representation, reconstructing decoupled clothed avatars from monocular videos or even single images by incorporating 2D generated data. While 2D generation methods can produce video sequences from given pose sequences, they struggle to maintain inter-frame consistency. We further decompose the inter-frame inconsistencies in generated data into geometric and appearance inconsistencies. Based on the 3D representation, we ensure the final reconstruction and rendering results are both 3D-consistent and temporally coherent. Compared to existing 3D garment transfer methods relying on multi-view reconstruction, or most 2D approaches that use diffusion models for image-based garment editing without novel view/pose synthesis, our method only needs monocular video or even a single image as input while achieving visually plausible cloth dynamics effects during virtual try-on and novel pose animation. This represents a significant advancement in achieving virtual try-on with visually plausible cloth dynamics with just a single camera.