Switching between touch and voice: factors influencing modality selection in multimodal systems

Users frequently encounter multimodal systems and make decisions regarding the most suitable modality for a given task. This study examines users' modality selection in multimodal systems, focusing on the features and menu structures of available modalities. Modality selection tasks that considered the number of touches, syllables per touch (S/T), and menu structure were presented to participants. Binomial logistic regression analysis revealed that S/T and menu structure significantly predicted modality selection. The voice modality was predominantly chosen when S/T ranged from 2 to 3 in non-hierarchical tasks and 4 to 5 in hierarchical tasks. In contrast, the number of touches did not significantly predict modality selection. Additionally, modality features and menu structure significantly influenced interaction effort, including both physical and mental dimensions. This study identified modality switching points by evaluating the interaction effort associated with voice and touch modalities.

Practitioner Summary: This study identifies key factors - syllables per touch and menu structure - affecting touch and voice modality preferences in multimodal systems. Hierarchical tasks significantly influence the switching point. The findings provide a framework for designing adaptive interfaces that enhance usability and reduce interaction effort.