集成的视觉变压器和闪光注意力,用于嘴唇对话生成GANAN
Qiong Yang1,2, Yuxuan Bai3, Feng Liu1
1School of Computer Science, Xi'an Polytechnic University, Xi'an, 710048, Shaanxi, China.
Scientific reports
|February 24, 2024
概括
这项研究介绍了FA-GAN,这是一种新的唇对口 (LTS) 模型,可以显著提高中文和英语语音识别的准确性. FA-GAN解决了口唇说话变化和关节运动建模的关键挑战,以改善沟通.
科学领域:
- 人工智能的人工智能
- 语音技术 语言技术
- 计算机视觉 计算机视觉
背景情况:
- 嘴唇对语音 (LTS) 技术正在迅速发展,在辅助技术和人机交互方面的应用也在迅速发展.
- 目前的LTS方法,通常是基于GAN的,难以准确地识别汉语,并将唇部运动与语音变异对齐.
- 局部和全球唇部运动的联合建模不足导致现有LTS系统中的视觉模糊性和图像表示性差.
研究的目的:
- 开发一种先进的口唇对语音生成模型,克服中文识别和唇部运动对齐方面的局限性.
- 从唇部运动来增强合成语音的准确性和质量.
- 提高LTS技术在通信和可访问性方面的实际应用性.
主要方法:
- 拟议的FA-GAN (闪电注意力GAN) 架构.
- 视觉和音频的单独编码与联合唇部运动建模,以改进语音识别.
- 集成多层Swin变压器,以增强图像表示.
- 实现一个层次的代生成器,用于高级语音生成.
- 包括一个闪光注意力机制来提高计算效率.
主要成果:
- 与现有架构相比,FA-GAN在中文和英语数据集上表现出优异的性能.
- 实现了43.19%的中文识别错误率,为此类型模型报告的最低值.
- 该模型有效地解决了口唇说话变异和关节唇部运动建模方面的挑战.
结论:
- FA-GAN代表了口唇对口语生成技术的重大进步.
- 拟议的方法提高了语音识别的准确性,特别是对于中文.
- 这项工作有助于提高语言障碍患者的沟通能力和生活质量.
相关概念视频
01:16Types Of Transformers
976
Transformers can provide desired voltages to a circuit by modifying the number of turns in the secondary windings.
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
If the ratio of the number of turns in the secondary winding to that of the primary winding is greater than one, then the transformer is said to be a step-up transformer. In a step-up transformer, the voltage at the secondary winding is greater than the voltage applied at the primary winding.
However, if this ratio is less than one, the transformer is said to be a step-down...
976
01:26The Ideal Transformer
393
In single-phase two-winding transformers, two windings are coiled around a magnetic core characterized by cross-sectional area A and magnetic permeability μ. A phasor current i1 enters the left winding while i2 exits the right winding, establishing the fundamental working of the transformer through electromagnetic principles.
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
Ampere's Law forms the basis of understanding the magnetic field within the transformer. It states that the integral of the magnetic field intensity's...
393