Investigating audiovisual integration of emotional signals in the human brain

Humans can communicate their emotional state via facial expression and affective prosody. This chapter reviews behavioural, neuroanatomical, electrophysiological and neuroimaging studies pertaining to audiovisual integration of emotional communicative signals. Particular emphasis will be given to neuroimaging studies using positron emission tomography (PET) or functional magnetic resonance imaging (fMRI). Conjunction analyses, interaction analyses, correlation analyses between haemodynamic responses and behavioural effects and connectivity analyses have been employed to analyse neuroimaging data. There is no general agreement as to which of these approaches can be considered "optimal" to classify brain regions as multisensory. We argue that these approaches provide complementing information as they assess different aspects of multisensory integration of emotional information. Assets and drawbacks of the different analysis types are discussed and demonstrated on the basis of one fMRI data set.