Creating Multimodal Interactive Digital Twin Characters From Videos: A Dataset and Baseline

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Digital twin characters require multimodal, character-specific data for realistic interaction.
• Existing datasets lack the scale and detail for complex character trait modeling.

Purpose of the Study:

• To develop a novel framework for creating multimodal interactive digital twin characters from dialogue videos.
• To introduce the Multimodal Character-Centric Conversation Dataset (MCCCD) for enhanced digital character training.
• To establish a baseline framework for generating consistent and character-specific digital human responses.

Main Methods:

• Collected the large-scale Multimodal Character-Centric Conversation Dataset (MCCCD) with detailed annotations.
• Proposed a baseline framework utilizing large language models for dialogue, speech synthesis for voice, and 3D talking head models for visuals.
• Enabled digital twins to replicate external characteristics (appearance, tone) and internal attributes (personality, behaviors).

Main Results:

• The MCCCD is approximately ten times larger per character than existing datasets.
• The proposed framework significantly outperforms existing methods in generating consistent and character-specific responses.
• Demonstrated the creation of digital twin characters capable of harmonious textual, vocal, and visual interaction.

Conclusions:

• The developed framework and MCCCD dataset establish a new benchmark for digital character creation.
• Paved the way for highly interactive and natural digital avatars with practical applications.
• Publicly released the dataset and code to foster further research in digital human technology.