VPT:视频肖像转换器为现实的说话面孔生成
Zhijun Zhang1, Jian Zhang2, Weijian Mai3
1School of Automation Science and Engineering, South China University of Technology, China; Key Library of Autonomous Systems and Network Control, Ministry of Education, China; Jiangxi Thousand Talents Plan, Nanchang University, China; College of Computer Science and Engineering, Jishou University, China; Guangdong Artificial Intelligence and Digital Economy Laboratory (Pazhou Lab), China; Shaanxi Provincial Key Laboratory of Industrial Automation, School of Mechanical Engineering, Shaanxi University of Technology, Hanzhong, China; School of Information Science and Engineering, Changsha Normal University, Changsha, China; School of Automation Science and Engineering, and also with the Institute of Artificial Intelligence and Automation, Guangdong University of Petrochemical Technology, Maoming, China; Key Laboratory of Large-Model Embodied-Intelligent Humanoid Robot (2024KSYS004), China; The Institute for Super Robotics (Huangpu), Guangzhou,, China.
这项研究引入了一种新的说话面孔生成方法,视频肖像转换器 (VPT),用于具有身份保护和自然眼的现实视频. 该框架改善了数字助理等应用程序的视听同步和面部细节.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 现有的以音频驱动的说话面部生成方法与摄影现实主义,身份保护和自然的面部细节 (如眼) 斗争.
- 音频和视频之间的同步是当前说话面部合成的一个关键挑战.
研究的目的:
- 提出一种新的说话面孔生成框架,视频肖像转换器 (VPT),解决现实主义,身份保护和眼同步的局限性.
- 为了增强与可控和自然的眼动作摄影现实的说话面部视频的合成.
主要方法:
- 拟议的视频肖像转换器 (VPT) 框架采用了两阶段的过程:音频到地标和地标面对面.
- 音频到地标阶段使用变压器编码器从音频和眼角比 (EAR) 预测面部地标.
- 里程碑式面对面的阶段使用视频到视频 (vid-to-vid) 网络进行里程碑式到现实的视频合成,包含自发闪生成模块.
主要成果:
- 通过VPT方法,成功地生成了具有高身份保护和精确的视听同步的摄影现实性说话面孔视频.
- 自发眼生成模块有效地模仿了真正的眼持续时间分布和频率,为生成的视频增加了自然性.
- 广泛的实验证实了该框架能够制作高质量的说话面部视频,并使用自然的眼动作.
结论:
- 视频肖像转换器 (VPT) 框架在音频驱动的说话面孔生成方面取得了重大进展,实现了卓越的现实主义和身份保护.
- 集成可控制的眼动作增强了合成说话面孔的自然性和表现力.
- 这项研究有助于在虚拟助理和视频会议等应用中实现更具身临其境和现实的数字交互.
更多相关视频
06:53Creating Virtual-hand and Virtual-face Illusions to Investigate Self-representation
Published on: March 1, 2017
06:20Author Spotlight: Development of an Automated Camera-Based System for Real-Time Blast Overpressure Monitoring and TBI Risk Assessment in Military Training
Published on: December 6, 2024
相关概念视频
Types Of Transformers
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
Masking and Demasking Agents
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on the metal...
Muscles for Facial Expressions
Transformers with Off-Nominal Turns Ratios
Facial Feedback Hypothesis
Transformation
