GSmoothFace:通过细粒度的3D面部指导生成通用的平滑说话面部
概括
GSmoothFace是一个新的说话面孔生成模型,合成了平滑的唇部动作,并且在没有avatar特定训练的情况下保留了身份. 这种通用方法提高了现实主义和口唇同步,为不同的扬声器.
科学领域:
- 计算机视觉 计算机视觉
- 人工智能的人工智能
- 语音处理 语音处理
背景情况:
- 现有的以语音驱动的说话面孔生成方法需要对化身进行特定训练,并与不稳定的唇部运动作斗争,从而限制了现实世界的应用.
- 当前的模型往往无法在不同的扬声器上进行概括,也无法在合成过程中保持身份保护.
研究的目的:
- 提出GSmoothFace,这是一个两阶段的通用谈话面孔生成模型,解决了现有方法的局限性.
- 为了实现准确而流的唇部同步,同时保持说话者的身份.
- 为了使不同扬声器的概括,而不需要再培训.
主要方法:
- 开发了一个两阶段模型:音频到表情预测 (A2EP) 和目标自适应面部翻译 (TAFT).
- A2EP模块使用变压器从音频中预测表情参数,从细粒度的3D面部顶部学习精确的唇同步.
- 通过形态增强面部混合 (MAFB) 增强的TAFT模块,使用预测参数修改目标视频面部,保持身份和背景.
主要成果:
- 与现有方法相比,在现实主义,唇同步和视觉质量方面取得了卓越的表现.
- 在不需要重新培训的情况下,向不同的发言者展示了有效的概括.
- 成功合成了平滑的唇部动态,同时保留了说话者的身份.
结论:
- GSmoothFace在通用的说话面孔世代中提供了显著的进步.
- 该模型的概括和保持身份的能力使其适合于现实世界的应用.
- 提出的方法克服了avatar特定训练和唇部运动稳定性的关键挑战.
相关概念视频
01:14Muscles for Facial Expressions
The craniofacial muscles are a collection of approximately 20 thin skeletal muscles situated beneath the skin of the face and scalp. These muscles, primarily responsible for the vast array of human facial expressions, originate from the bones or fibrous structures of the skull and extend outwards to connect with the skin. While most skeletal muscles in the body are enveloped in thick fascia, facial muscles generally have a more delicate fascial covering, with the buccinator muscle being a...
01:25Shape and Texture of Coarse Aggregate
Aggregate shape is classified based on the relative sharpness or roundness of the edges and corners. This classification includes categories like rounded, angular, elongated, and flaky, each with specific characteristics. Rounded aggregates, fully shaped by attrition, are typical of river or seashore gravel, while angular aggregates, such as crushed rock, have well-defined edges. Aggregates that are elongated and flaky are less desirable, as they can reduce the workability and strength of...
01:19Tangent Planes to Surfaces
In multivariable calculus, the concept of a tangent plane plays a central role in approximating curved surfaces. When dealing with a surface defined by a function of two variables, such as z = f(x, y), the tangent plane at a given point provides the best linear approximation to the surface near that point. This local linearization allows complex, nonlinear geometries to be treated using simpler, planar models.The construction of the tangent plane involves taking vertical slices of the surface...
01:31Tangent Planes to Level Surfaces
A level surface consists of all points in space where a function of three variables takes the same fixed value. If a point lies on this surface, understanding the surface’s geometry there requires more than just knowing the point’s coordinates; it requires describing how the surface is oriented, or how it tilts, near that point.To probe this local geometry, imagine tracing a path that stays entirely on the level surface and passes through the point of interest. This path can be described as a...
01:22Tangent Planes to a Parametric Surface
A tangent plane provides a linear approximation to a curved surface at a specific point, capturing the local behavior of the surface. It can be understood as the plane that just touches the surface at that point and is defined by the tangent directions of curves lying on the surface. These tangent directions arise naturally when the surface is described parametrically, allowing systematic construction of the plane.For a surface expressed in parametric form, the position of any point is...
01:30Oriented Surfaces
A surface is called orientable if a consistent choice of unit normal vector can be made at every point on the surface. A thin soap film stretched across a wire loop provides a familiar example. The film separates the air on one side from the air on the other, so one side can be selected as positive and the opposite side as negative. Once this choice is made, a unit normal vector can be assigned smoothly across the entire surface.At each point on the soap film, a unit normal vector points...