Audiovisual investigation of the loudness-effort effect for speech and nonspeech events

There is some evidence that loudness judgments of speech are more closely related to the degree of vocal effort induced in speech production than to the speech signal's surface-acoustic properties such as intensity. Other researchers have claimed that speech loudness can be rationalized simply by considering the acoustic complexity of the signal. Because vocal effort can be specified optically as well as acoustically, a study to test the effort-loudness hypothesis was conducted that used conflicting audiovisual presentations of a speaker that produced consonant-vowel syllables with different efforts. It was predicted that if loudness judgments are constrained by effort perception rather than by simple acoustic parameters, then judgments ought to be affected by visual as well as auditory information. It is shown that loudness judgments are affected significantly by visual information even when subjects are instructed to base their judgments only on what they hear. A similar (though less pronounced) patterning of results is shown for a nonspeech "clapping" event, which attests to the generality of the loudness-effort effect previously thought to be special to speech. Results are discussed in terms of auditory, fuzzy logical, motor, and ecological theories of speech perception.