Flexible Visuomotor Associations in Touchscreen Control

To move real objects, our hand needs to get in direct physical contact with the object. However, this is not necessarily the case when interacting with virtual objects, for example when displacing objects on tablets by swipe movements. Here, we performed two experiments to study the behavioral strategies of these movements, examining how visual information about the virtual object is mapped into a swipe that moves the object into a goal location. In the first experiment, we investigated how swiping behavior depends on whether objects were located within or outside the swiping workspace. Results show that participants do not start the swipe movement by placing their finger on the virtual object, as they do when reaching to real objects, but rather keep a systematic distance between the object location and the initial swipe location. This mismatch, which was experimentally imposed by placing the object outside the workspace, also occurred when the object was within the workspace. In the second experiment, we investigated which factors determine this mismatch by systematically manipulating the initial hand location, the location of the object and the location of the goal. Dimensionality reduction of the data showed that three factors are taken into account when participants choose the initial swipe location: the expected total movement distance, the distance between their finger on the screen and the object, and a preference not to cover the object. The weight given to each factor differed among individuals. These results delineate, for the first time, the flexibility of visuomotor associations in the virtual world.