达甘++:深度意识的生成对抗网络用于说话头部视频生成
IEEE transactions on pattern analysis and machine intelligence
|December 6, 2023
概括
本研究介绍了一种自我监督的方法,通过从视频中学习密集的3D面部几何学来生成现实的3D说话头视频. 这种新的方法可以实现最先进的结果,而不需要3D注释.
科学领域:
- 计算机视觉 计算机视觉
- 计算机图形 计算机图形
- 机器学习 机器学习
背景情况:
- 目前的说话头生成方法主要使用2D面部信息,限制了3D准确性.
- 密集的3D面部几何,像像素智能深度一样,对于精确的3D重建和噪音抑制至关重要.
- 获得面部视频密集的3D注释是昂贵和具有挑战性的.
研究的目的:
- 从没有3D注释的视频中开发一种自我监督的方法来学习密集的3D面部几何 (深度).
- 为了提高面部关键点估计的准确性,用于运动场生成.
- 为了创建一个3D意识的注意力机制,以增强面部几何学捕捉在说话头合成.
主要方法:
- 从面部视频中学习密集的3D面部几何 (深度) 的新的自我监督方法,消除了对摄像机参数或3D注释的需求.
- 一种学习像素级不确定性的策略,用于识别用于几何学习的可靠像素.
- 一个以几何指导的面部关键点估计模块,用于准确的运动场生成.
- 一个具有3D意识的交叉模式注意力机制,整合了外观和深度,以捕捉粗细的面部几何形状.
主要成果:
- 拟议的框架成功地产生了极具现实性的重现的交谈视频.
- 在VoxCeleb1,VoxCeleb2和HDTF基准标准上取得了新的最先进的性能.
- 该方法有效地利用自我监督学习来进行密集的3D面部几何提取.
结论:
- 开发的框架为3D说话头的生成提供了具有成本效益和高效的解决方案.
- 自主监督的3D面部几何学的学习显著提高了产生的视频的现实性和准确性.
- 几何指导的关键点和跨模态注意力的集成推进了说话头合成的最先进技术.
相关概念视频
Muscles that Move the Head
1.9K
The muscles that move the head are a dynamic and complex group of structures that work together to facilitate a wide range of head movements, including rotation, flexion, extension, and lateral bending.
The bilateral sternocleidomastoid, or SCM, and the suprahyoid and infrahyoid muscles are significant head flexors. The SCM muscles originate at the sternum and clavicle and attach to the mastoid process of the temporal bone. The SCM contracts bilaterally to bend the head forward, whereas...
The bilateral sternocleidomastoid, or SCM, and the suprahyoid and infrahyoid muscles are significant head flexors. The SCM muscles originate at the sternum and clavicle and attach to the mastoid process of the temporal bone. The SCM contracts bilaterally to bend the head forward, whereas...
1.9K
Non-equilibrium in the Cell
4.4K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
4.4K
Nonconscious Mimicry
4.6K
Nonconscious mimicry occurs when individuals alter their mannerisms to match the behaviors and expressions of those nearby, without intention.
4.6K
Masking and Demasking Agents
2.4K
EDTA titrations may necessitate masking and demasking agents to temporarily protect a particular metal ion in a mixture from the EDTA reaction. These agents facilitate the sequential analysis of the metal ions by forming stable complexes with some—but not all—metal ions during certain steps.
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
There are many masking agents, such as cyanide, fluoride, triethanolamine, thiourea, and 2,3-bis(sulfanyl)propan-1-ol (formerly 2,3-dimercapto-1-propanol), with the masking agent chosen based on...
2.4K


