Robust Manipulation of Randomly Stacked Jenga Blocks via a Strategy-Driven Framework Using a Single RGB-D Sensor

Robust manipulation of small, densely stacked objects remains a challenging problem due to severe occlusions and geometric ambiguities, particularly under single-view sensing conditions. When observed using a single RGB-D sensor, adjacent surfaces of featureless cuboid objects, such as Jenga blocks, often merge in depth measurements, making reliable instance separation and pose estimation difficult. This paper presents a strategy-driven perception and manipulation framework for the robotic rearrangement of randomly stacked Jenga blocks under single RGB-D sensor constraints. The proposed approach employs a heightmap-based perception pipeline that integrates color filtering with geometric reasoning to segment individual blocks and estimate manipulation-compatible poses. Beyond perception, the proposed system determines robot actions through a structured manipulation policy consisting of region-wise search for directly executable grasps, grasp candidate evaluation based on accessibility and collision risk, selective local regrasping for workspace reconfiguration, and placement mode selection between direct insertion and sliding-assisted placement. In this framework, controlled grasp-and-release actions are applied only when no directly executable candidate is found within the currently scanned region and a suitable recovery target can be identified, thereby transforming cluttered local arrangements into more executable states without requiring additional sensing modalities. Experimental results, conducted under competition-equivalent conditions, demonstrate a high task success rate of 99.02%, confirming the robustness and reliability of the proposed framework. The results show that strategy-driven manipulation can effectively compensate for perception limitations in single RGB-D sensor environments, enabling stable and efficient pick-and-place operations in dense clutter.