Transinger: Cross-Lingual Singing Voice Synthesis via IPA-Based Phonetic Alignment

Area of Science:

• Artificial Intelligence
• Speech Technology
• Computational Linguistics

Background:

• Singing Voice Synthesis (SVS) faces challenges with global linguistic diversity.
• Fragmented, language-specific phoneme encodings hinder unified phonetic modeling in SVS.
• Existing SVS research lacks exploration of cross-lingual generalization.

Purpose of the Study:

• To develop a cross-lingual singing voice synthesis framework that overcomes linguistic barriers.
• To improve phonetic representation and generalization capabilities in SVS models.
• To enable SVS for languages not explicitly included in the training data.

Main Methods:

• Created a four-language dataset using International Phonetic Alphabet (IPA) for consistent phonetic representation.
• Proposed a novel method of decomposing IPA phonemes into letters and diacritics for deeper pronunciation rule learning.
• Introduced Transinger, a cross-lingual synthesis framework integrating Conformer and RVQ techniques.
• Developed a dynamic IPA adaptation strategy for applying learned representations to unseen languages.

Main Results:

• Transinger demonstrated significant improvements in cross-lingual generalization compared to state-of-the-art methods.
• Objective and subjective experiments confirmed the effectiveness of the proposed approach.
• The method achieved outstanding cross-lingual synthesis performance, even for languages not seen during training.

Conclusions:

• Multilingual aligned representations enhance SVS model learning efficacy and robustness.
• Decomposing IPA phonemes into letters and diacritics improves pronunciation learning and generalization.
• Transinger represents a breakthrough in cross-lingual singing voice synthesis.