Bidirectional coupling of physiology and emotions reveals unique speaker-listener dynamics

Conversational interactions extend beyond verbal exchange, involving dynamic synchronization across physiological, emotional, and acoustic domains. This study aimed to characterize multimodal speaker-listener coupling during structured debates to examine how bidirectional dynamics relate to autonomic regulation, signal complexity, and emotional states. We analyzed data from the K-EmoCon database, which included 32 participants in 10-minute debates. Heart rate variability (HRV) features, speech features, and multi-perspective emotion annotations were synchronized and analyzed in segmented speak-listen phases. Cross-correlation and bidirectional coupling quantified coupling strength and directionality, and their relationships with emotional states were assessed. Negative lag segments showed significantly higher HRV features, including AVNN (0.655 [0.631-0.749], P = .003), HF power (0.620 [0.338-0.781], P < .001), and SD1 (1.30 × 10-3 [1.19-1.41] × 10-3, P = .003). Positive lag segments were associated with higher sample entropy (0.058 [0.046-0.065], P < .001) and diffusion entropy (μr: 1.209 [1.201-1.221], P < .001). Lower emotion states (1-2) tended to exhibit negative-lag dynamics, whereas moderate to high states (3 and 5) showed a modest positive-lag predominance. Multimodal coupling reveals distinct physiological and emotional signatures linked to leadership and responsiveness in conversation, providing insights for therapeutic, educational, and collaborative applications.