Speechreading supplemented by single-channel and multichannel tactile displays of voice fundamental frequency

The benefits of two tactile codes of voice fundamental frequency (F0) were evaluated as supplements to the speechreading of sentences in two short-term training studies, each using 12 adults with normal hearing. In Experiment 1, a multichannel spatiotemporal display of F0, known as Portapitch, was used to stimulate the index finger. In an attempt to improve on past performance with this display, the coding scheme was modified to better cover the F0 range of the talker in the training materials. For Experiment 2, to engage kinesthetic/proprioceptive pathways, a novel single-channel positional display was built, in which F0 was coded as the vertical displacement of a small finger-rest. Input to both displays consisted of synthesized replicas of the F0 contours of the sentences, prepared and perfected off-line. Training with the two tactile F0 displays included auditory presentation of the synthesized F0 contours in conjunction with the tactile patterns on alternate trials. Speechreading enhancement by the two tactile F0 displays was compared to the enhancement provided when auditory F0 information was available in conjunction with the tactile patterns, by auditory presentation of a sinusoidal indication of the presence or absence of voicing, and by a single-channel tactile display of the speech waveform presented to the index finger. Despite the modified coding strategy, the multichannel Portapitch provided a mean tactile speechreading enhancement of 7 percentage points, which was no greater than that found in previous studies. The novel positional F0 display provided only a 4 percentage point enhancement. Neither F0 display was better than the simple single-channel tactile transform of the full speech waveform, which gave a 7 percentage point enhancement effect. Auditory speechreading enhancement effects were 17 percentage points with the voicing indicator and approximately 35 percentage points when the auditory F0 contour was provided in conjunction with the tactile displays. The findings are consistent with the hypothesis that subjects were not taking full advantage of the F0 variation information available in the outputs of the two experimental tactile displays.