Blending speech output and visual text in the multimodal interface

Simultaneous reading and listening with a redundant display of visual text with speech output was investigated to determine how variations in verbal working memory capacity and content complexity affected comprehension.

Previous work has found some evidence of a benefit for displays that blend speech and visual text; content complexity and verbal working memory capacity are likely to significantly determine this benefit.

In the experiment reported here, a multimodal display of e-mail messages was compared with speech output alone and with a purely visual display. Comprehension of the messages was examined in relation to verbal working memory capacity and the complexity of the messages. Thirty-two users participated in the study, which used a repeated measures design.

The data show that the multimodal interface did not affect comprehension relative to a purely visual interface, even when the content was more complex, although it did improve the comprehension of complex information relative to a purely auditory interface. Lower-capacity participants were neither especially advantaged nor disadvantaged by the multimodal interface. Participants expressed a marked preference for the multimodal display of the more complex sentences.

The experiment suggests that a redundant multimodal display will neither assist nor disrupt understanding when compared with a purely visual display, but it will assist understanding of complex content when compared with speech output alone.

Redundant displays of visual text and speech have potential application in multitask situations, in multimedia presentations, and for devices with small screens.