Synthesizing vocal tract magnetic resonance imaging sequences with phoneme-aware diffusion models

Area of Science:

• Medical Imaging
• Speech Science
• Artificial Intelligence

Background:

• Real-time speech Magnetic Resonance Imaging (MRI) provides crucial data for speech diagnostics, therapy, and scientific research.
• Directly mapping speech to MRI remains a significant technical challenge, limiting current applications.

Purpose of the Study:

• To develop a diffusion-based framework for synthesizing dynamic vocal tract MRI sequences directly from spoken language.
• To enhance phoneme-level fidelity in speech-to-MRI mapping.
• To introduce a new metric, the Speech-to-Image Phonemic Fidelity Score (SIPFS), for assessing articulatory precision.

Main Methods:

• Trained audio-conditioned diffusion models with phonemic-aware attention and contrastive learning on a large-scale speech MRI database.
• Utilized pre-trained audio encoders (Wav2Vec, WavLM, Whisper) for speech embedding.
• Evaluated performance using standard image/video metrics (FID, FVD, SSIM, PSNR) and the novel SIPFS on unseen speakers and speech segments.

Main Results:

• Phonemic-aware attention significantly improved image quality and phoneme-level precision.
• The proposed framework demonstrated robust generalization to new speakers and speech segments.
• SIPFS indicated up to 15% higher phonemic accuracy compared to unconditioned models, validating the approach.

Conclusions:

• The developed framework successfully synthesizes accurate, phoneme-sensitive vocal tract MRI from speech using diffusion models.
• This advancement supports clinical applications in speech diagnostics, therapy, and articulatory research.
• Future work aims to expand datasets, improve computational efficiency, and refine evaluation metrics for broader clinical integration.