Robust Manipulation of Randomly Stacked Jenga Blocks via a Strategy-Driven Framework Using a Single RGB-D Sensor

Area of Science:

• Robotics
• Computer Vision
• Artificial Intelligence

Background:

• Manipulating densely stacked, featureless objects like Jenga blocks is difficult due to occlusions and depth ambiguities from single RGB-D sensors.
• Existing methods struggle with reliable instance separation and pose estimation in cluttered environments.

Purpose of the Study:

• To develop a strategy-driven perception and manipulation framework for robotic rearrangement of Jenga blocks under single RGB-D sensor constraints.
• To address limitations in perception by integrating color filtering and geometric reasoning for block segmentation and pose estimation.

Main Methods:

• A heightmap-based perception pipeline combining color filtering and geometric reasoning.
• A structured manipulation policy including region-wise grasp search, grasp evaluation, local regrasping, and placement mode selection.
• Controlled grasp-and-release actions to recover from cluttered states without additional sensors.

Main Results:

• Achieved a 99.02% task success rate in experiments under competition-equivalent conditions.
• Demonstrated robustness and reliability in pick-and-place operations within dense clutter.
• Showcased effective compensation for perception limitations using strategy-driven manipulation.

Conclusions:

• The proposed framework enables stable and efficient robotic manipulation in challenging single RGB-D sensor environments.
• Strategy-driven manipulation is a viable approach to overcome perception limitations in dense object arrangements.
• The system effectively transforms cluttered local arrangements into more executable states.