Switching between touch and voice: factors influencing modality selection in multimodal systems

Area of Science:

• Human-Computer Interaction
• Multimodal Systems Design

Background:

• Users navigate multimodal systems, selecting between various input modalities.
• Understanding modality selection is crucial for optimizing user experience in interactive systems.

Purpose of the Study:

• To investigate factors influencing user modality selection in multimodal systems.
• To analyze the impact of syllables per touch (S/T) and menu structure on modality choice.
• To determine how modality features and menu structure affect interaction effort.

Main Methods:

• Participants completed modality selection tasks involving varying S/T and menu structures.
• Binomial logistic regression was used to analyze modality selection predictors.
• Interaction effort (physical and mental) was evaluated to identify modality switching points.

Main Results:

• Syllables per touch (S/T) and menu structure significantly predicted modality selection.
• Voice modality was preferred for specific S/T ranges depending on task hierarchy (non-hierarchical: 2-3 S/T; hierarchical: 4-5 S/T).
• Number of touches did not significantly predict modality selection, but modality features and menu structure influenced interaction effort.

Conclusions:

• S/T and menu structure are key determinants of modality preference in multimodal interfaces.
• Task hierarchy influences the optimal S/T for voice modality selection.
• Interface design should consider S/T and menu structure to minimize interaction effort and enhance usability.