An 18-DOF hand integrating force-position multimodal perception using a monocular camera

The anthropomorphic hand plays a crucial role in human-machine interaction tasks. However, there are very few hands that realize multimodal perception with high degrees of freedom (DOF) in a low-cost way. Here, we present a dexterous hand that achieves multimodal sensing solely through a camera. The hand has 18 DOF but does not require any position or force sensors, making it cost-effective and easy to manufacture. We develop an integrated forearm for the hand that provides both actuation and multimodal sensing information simultaneously. This includes the 18 joint angles, 5 fingertip positions and contact forces, and information on object softness and contour. The core principle of perception is that the camera can track the displacement and tension of all tendons simultaneously. The multimodal perception model is developed by characterizing tendon properties and coupling them with the hand dynamics. Experiments indicate that our hand has potential in multimodal sensing and dexterity.