Sketch2HOI: Sketch-Based Modeling of 3D Hand-Object Interaction with Hierarchical Attentive Fusion

Accurate modeling of 3D hand-object interactions plays a crucial role in modern 3D animation, serving as an essential component in animated movies and video game development. Traditional workflows require artists to manually adjust hand joints to create hand-object interactions, which is labor intensive and inefficient. We propose a novel framework for modeling 3D hand-object interactions from free-hand sketches. Our method generates physically plausible hand poses while ensuring precise alignment with user inputs. To achieve this, we introduce a pipeline that attentively integrates 2D sketch and 3D object information to model dexterous hand interactions. The core design principle is effective fusion of 2D and 3D features with explicit awareness of hand-object contact relationships. The overall framework mainly consists of three components, including a voxel-based feature fusion module for multi-modal representation, a hierarchical attentive prediction module for modeling long-range dependencies while capturing fine-grained details and a graph-based contact-aware adjustment module for refinement. Our method achieves strong performance and generalizes well to novel object categories. These components together yield a robust, user-friendly sketch-based 3D hand object interaction modeling system. Extensive experiments have verified the superiority of our proposed method compared with state-of-the-art works.