The combined perception of emotion from voice and face: early interaction revealed by human electric brain responses

Judgement of the emotional tone of a spoken utterance is influenced by a simultaneously presented face expression. The time course of this integration was investigated by measuring the mismatch negativity (MMN). In one condition, the standard stimulus was an angry voice fragment combined with a (congruous) angry face expression. In the deviant pair, the voice expression was kept the same and only the face expression changed to an (incongruous) sad face. The pairs with a deviant visual item evoked a negative electric brain response showing the characteristics of the MMN, which is usually evoked only by auditory deviations. Similar results were obtained by employing incongruous standard and congruous deviant pairs. These findings provide compelling evidence of an early integration of face with voice information in the processing of affect.