Tri-attention complex-valued CNN for multimodal holographic digital human interaction.

Area of Science:

• Computer Vision
• Holography
• Artificial Intelligence

Background:

• Holographic displays offer immersive AR/VR experiences but face challenges in full-color generation, visual consistency, and real-time interaction.
• Existing systems struggle with high-fidelity holographic reconstruction and seamless user interaction.

Purpose of the Study:

• To propose a multimodal voice-interactive holographic digital human avatar system.
• To enhance the quality of full-color computer-generated holography (CGH) and enable real-time interaction.

Main Methods:

• Utilized an RGB-D digital human avatar generated by AI-generated content (AIGC) technology for personalized configurations.
• Implemented an end-to-end semantic interaction architecture ('ASR + Dual-LLM + TTS') for multi-agent voice interaction.
• Developed a Tri-Attention Complex-valued Convolutional Neural Network (TA-CCNN) with an Energy-Aware branch for high-fidelity holographic reconstruction.

Main Results:

• Achieved end-to-end RGB-D generation with personalized avatar configurations (outfit, personality, voice cloning).
• Enabled high-quality, real-time voice interaction through the multi-agent system.
• Significantly improved contrast in high-energy regions, enhancing color reproduction and visual effects with the TA-CCNN.

Conclusions:

• The proposed system successfully integrates voice interaction with high-fidelity holographic avatars.
• The TA-CCNN advances full-color holographic reconstruction, offering superior visual quality for immersive applications.
• This work paves the way for more interactive and visually compelling holographic experiences in AR/VR.