基于Whisper细分的实时多语言语音识别和扬声器日记系统.
Ke-Ming Lyu1, Ren-Yuan Lyu1, Hsien-Tsung Chang1,2,3
1Computer Science and Information Engineering, Chang Gung University, Taoyuan, Taiwan.
PeerJ. Computer science
|April 25, 2024
概括
本研究介绍了一个实时的多语言语音系统,使用OpenAI的Whisper模型进行准确的语音日记化 (SD) 和自动语音识别 (ASR),即使有口音和语音变化.
科学领域:
- 语音处理 语音处理
- 人工智能的人工智能
- 自然语言处理自然语言处理.
背景情况:
- 传统的语音识别和说话者日记化系统在动态的多语言环境中扎,尤其是口音和频繁的说话者变化.
- 在当前的系统中,准确处理普通话语的语音与台湾口音以及管理发言者的转换是重大挑战.
研究的目的:
- 开发一套尖端的,实时的多语言语音识别和演讲者日记系统.
- 在复杂的多音箱场景中提高性能,重点是用台湾口音和快速音箱转换的普通话.
主要方法:
- 利用 OpenAI 的 Whisper 模型来实现核心语音识别功能.
- 集成先进的扬声器日记化技术,包括高效的输出处理和扬声器嵌入技术,用于实时应用.
- 优化了系统的动态,多扬声器环境与频繁的扬声器开关.
主要成果:
- 实现了一个有前途的单词日记化错误率 (WDER) 总体为6.96%.
- 在双扬声器 (2.68% WDER) 和三扬声器 (11.65% WDER) 场景中表现出强的性能.
- 实时性能与非实时基线模型可比.
结论:
- 开发的系统代表了实时多语言语音处理的重大进步.
- 该系统有效地处理复杂的对话动态,包括不同的口音和多个扬声器.
- 这项研究验证了将先进的ASR和SD技术集成到现实应用中的有效性.
相关概念视频
01:21Perceiving Loudness, Pitch, and Location
208
The human brain perceives pitch through two primary mechanisms reflected in place theory and frequency theory. Each mechanism describes how sound waves are interpreted as specific pitches by the brain, offering insights into the intricate processes of auditory perception.
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
Place theory, or place coding, suggests that different pitches are heard because various sound waves activate specific locations along the cochlea's basilar membrane. The brain determines the pitch of a sound by...
208
01:22Chunking and Rehearsal in Sensory Memory
202
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
202