OmniCharacter++: Towards Comprehensive Benchmark for Realistic Role-Playing Agents.

Area of Science:

• Artificial Intelligence
• Natural Language Processing
• Human-Computer Interaction

Background:

• Current role-playing agents (RPAs) are evaluated in limited, text-only, two-person conversations.
• This fails to capture the complexity of real-world human interactions involving multiple people and various communication modes.

Purpose of the Study:

• Introduce OmniCharacter++, the first benchmark for assessing multi-character interactions in a combined text and speech context.
• Provide a large-scale dataset and evaluation suite for multi-modal role-playing scenarios.
• Develop and evaluate a unified text-speech model for managing complex multi-character dynamics.

Main Methods:

• Developed OmniCharacter++, a dataset with 10,287 characters, 118,017 dialogues, and over 1 million audio responses across 8 topics.
• Created a comprehensive evaluation suite assessing dialogue understanding, generation quality, and perceptual naturalness.
• Trained UniCharacter-7B, a unified text-speech model on the OmniCharacter++ dataset.

Main Results:

• UniCharacter-7B demonstrated more realistic and consistent role-playing responses, enhancing both attractiveness and consistency.
• The OmniCharacter++ benchmark presents significant challenges for current state-of-the-art models.
• The study establishes a clear direction for future research in multi-modal, multi-character AI.

Conclusions:

• OmniCharacter++ advances the evaluation of role-playing agents beyond static, text-based interactions.
• UniCharacter-7B shows promise in handling complex multi-character dynamics with improved vocal and semantic alignment.
• This work paves the way for more sophisticated and human-like conversational AI systems.