UniEmo: Unifying Emotional Understanding and Generation With Learnable Expert Queries

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Emotional understanding and generation are typically separate research areas.
• These tasks are complementary and can mutually benefit from integration.
• Existing methods lack a unified approach for joint emotional processing.

Purpose of the Study:

• To propose UniEmo, a unified framework integrating emotional understanding and generation.
• To address the challenge of abstract emotional representations with visual features.
• To enhance the performance of both emotional understanding and generation tasks.

Main Methods:

• Developed a hierarchical emotional understanding chain with learnable expert queries.
• Fused expert queries and emotional representations to guide a diffusion model for image generation.
• Introduced emotional correlation coefficient and emotional condition loss for improved generation.
• Implemented a novel data filtering algorithm for quality feedback.
• Utilized joint training with dual feedback loops between generation and understanding.

Main Results:

• UniEmo significantly outperforms state-of-the-art methods in both emotional understanding and generation.
• The hierarchical understanding chain effectively extracts multi-scale emotional features.
• Generation-driven dual feedback processes enhance the model's understanding capacity.
• The framework generates diverse and high-fidelity emotion-evoking images.

Conclusions:

• UniEmo provides a powerful unified framework for emotional AI tasks.
• The proposed methods effectively bridge the gap between emotional understanding and generation.
• The dual feedback mechanism is crucial for enhancing model performance.
• This work opens new avenues for research in multimodal emotion AI.