AI-driven audio-to-video generation for dynamic content creation via stable diffusion and CNN-augmented transformers.

Area of Science:

• Generative Artificial Intelligence (AI)
• Computer Vision
• Human-Computer Interaction

Background:

• Generating emotionally resonant video from speech is challenging for AI.
• Existing models misinterpret vocal cues, leading to disconnected visuals.
• Subtle speech patterns often obscure emotional context.

Purpose of the Study:

• Introduce EchoVid, an audio-to-video synthesis model.
• Prioritize contextual fidelity and emotional alignment in AI-generated video.
• Address limitations in current speech-to-video generation.

Main Methods:

• Utilized PyAudio for speech input and Hugging Face's Stable Diffusion v2.1.
• Employed emotion-aware prompts and CNN-enhanced diffusion transformers.
• Developed a web interface (React.js, TypeScript, Node.js, MongoDB Atlas) for interaction.

Main Results:

• EchoVid generates videos reflecting emotional tone and context.
• Outperformed MoCoGAN and Stable Video Diffusion variants.
• Introduced novel metrics: Temporal Semantic Stability (TSS) and Perceptual Flicker Index (PFI).

Conclusions:

• EchoVid demonstrates superior performance in audio-to-video synthesis.
• Achieves better emotional and contextual alignment compared to existing methods.
• Novel metrics provide enhanced evaluation for video generation quality.