从视频创建多模式交互式数字双胞胎字符:数据集和基线
IEEE transactions on pattern analysis and machine intelligence
|August 28, 2025
概括
这项研究介绍了从电视节目中创建互动式数字双胞胎角色的框架, 新的数据集和方法推动了现实的数字化人类创造.
科学领域:
- 人工智能
- 计算机视觉
- 自然语言处理
背景情况:
- 数字双胞胎字符需要多模式,字符特定的数据来实现真实的交互.
- 现有的数据集缺乏复杂的特征建模的规模和细节.
研究的目的:
- 从对话视频中创建多式交互式数字双胞胎角色的新框架.
- 引入多式型以角色为中心的对话数据集 (MCCCD),以加强数字角色培训.
- 建立一个基准框架来产生一致和特异性的数字人类反应.
主要方法:
- 收集了大规模的多模态以角色为中心的对话数据集 (MCCCD),并提供了详细的注释.
- 提出了使用大型语言模型进行对话的基线框架,语音合成和3D说话头模型用于视觉.
- 使数字双胞胎能够复制外部特征 (外观,语气) 和内部属性 (个性,行为).
主要成果:
- 每个字符的MCCCD大约是现有的数据集的十倍.
- 提出的框架在产生一致和特征特定的响应方面显著优于现有方法.
- 展示了能够协调文本,语音和视觉交互的数字双胞胎字符的创建.
结论:
- 开发的框架和MCCCD数据集为数字字符创建建立了一个新的基准.
- 开辟了高度互动和自然的数字化身的道路.
- 公开发布数据集和代码,以促进数字人类技术的进一步研究.
相关概念视频
Modeling and Similitude
328
Scaled modeling is a fundamental technique in engineering, enabling the study of large and complex systems by creating smaller, manageable replicas that recreate critical characteristics of the original. In hydrology and civil infrastructure, for example, scaled models of dams help analyze water flow, turbulence, and pressure. This method allows for accurate predictions of real-world behavior within a controlled environment, significantly reducing the cost and time involved in full-scale...
328
Nonconscious Mimicry
4.6K
Nonconscious mimicry occurs when individuals alter their mannerisms to match the behaviors and expressions of those nearby, without intention.
4.6K
Muscles for Facial Expressions
2.7K
The craniofacial muscles are a collection of approximately 20 thin skeletal muscles situated beneath the skin of the face and scalp. These muscles, primarily responsible for the vast array of human facial expressions, originate from the bones or fibrous structures of the skull and extend outwards to connect with the skin. While most skeletal muscles in the body are enveloped in thick fascia, facial muscles generally have a more delicate fascial covering, with the buccinator muscle being a...
2.7K
Composite Bodies
1.1K
A composite body is a body made up of multiple parts, connected to form a larger, unified object. Each part has its own weight and center of gravity, which must be considered to determine the center of gravity of the composite body. In cases where the density or specific weight is constant, the center of gravity coincides with the centroid.
Composite bodies have widespread applications in mechanical engineering, from automobiles to aircraft to rockets. For example, an automobile wheel comprises...
Composite bodies have widespread applications in mechanical engineering, from automobiles to aircraft to rockets. For example, an automobile wheel comprises...
1.1K
Facial Feedback Hypothesis
244
Charles Darwin proposed that facial expressions are an evolutionary adaptation for communication. He argued that these expressions are not influenced by culture but are universal across species. For example, a snarling expression with exposed teeth signals a threat in many animals, including humans. Darwin also suggested that displaying an emotion can intensify the feeling. Smiling, for example, could enhance one's sense of happiness. This idea laid the foundation for understanding the role...
244


