Learning to Generate Context-Sensitive Backchannel Smiles for Embodied AI Agents with Applications in Mental Health...

Area of Science:

• Human-Computer Interaction
• Artificial Intelligence
• Psychology

Background:

• A critical shortage of mental health resources necessitates innovative solutions for accessible therapeutic support.
• Embodied agents offer a promising, cost-effective supplement to traditional caregiving.
• Simulating non-verbal behaviors, like backchannels, is crucial for rapport in therapeutic contexts but is under-explored.

Purpose of the Study:

• To improve the rapport-building capabilities of embodied agents by simulating backchannel smiles.
• To investigate the predictors of backchannel smile duration and intensity in human conversations.
• To develop and evaluate a generative model for producing realistic backchannel smiles in embodied agents.

Main Methods:

• Annotated backchannel smiles in face-to-face conversations about sensitive topics.
• Analyzed speech prosody, language, and demographics as predictors of smile intensity.
• Developed an attention-based generative model incorporating listener information for smile production.

Main Results:

• Speaker and listener behaviors, including speech cues and demographics, significantly predict backchannel smile intensity.
• Listener information improved the performance of the generative model over speaker-centric approaches.
• Conditioned generation based on predictors significantly enhanced empirical measures of generation quality.

Conclusions:

• Backchannel smile production can be effectively modeled as a generation problem for embodied agents.
• Embodied agents exhibiting backchannel smiles are perceived as more human-like.
• Agents with simulated backchannel smiles present an attractive alternative for non-personal conversations, enhancing user experience.