RSATalker: Realistic Socially-Aware Talking Head Generation for Multi-Turn Conversation

Area of Science:

• Computer Vision
• Virtual Reality
• Artificial Intelligence

Background:

• Existing talking head generation methods in VR have limitations, including poor realism in 3D or high computational costs in 2D.
• 3D Gaussian Splatting (3DGS) offers efficient rendering but typically focuses on single speakers, neglecting social context.

Purpose of the Study:

• To introduce RSATalker, the first framework for realistic and socially-aware talking head generation using 3DGS.
• To enable multi-turn conversations with high-fidelity avatar rendering and accurate interpersonal dynamics.

Main Methods:

• Leveraging 3D Gaussian Splatting (3DGS) by binding Gaussians to mesh facets for rendering.
• Developing a socially-aware module to encode relationship dynamics (e.g., familial, hierarchical) into embeddings.
• Utilizing a three-stage training paradigm and the new RSATalker dataset with annotated social relationships.

Main Results:

• RSATalker achieves state-of-the-art performance in generating realistic talking head avatars.
• The framework successfully incorporates social awareness into avatar generation, capturing interpersonal nuances.
• Demonstrated effectiveness in applications like VR telepresence and embodied conversational agents.

Conclusions:

• RSATalker advances talking head generation by integrating 3DGS with social relationship modeling.
• The proposed socially-aware conditioning can be generalized to other human motion generation tasks.
• The framework enhances the immersion and realism of social interactions in virtual environments.