CMRI2SPEC: CINE MRI SEQUENCE TO SPECTROGRAM SYNTHESIS VIA A PAIRWISE HETEROGENEOUS TRANSLATOR.

Area of Science:

• Medical Imaging
• Acoustics
• Machine Learning

Background:

• Multimodal representation learning integrates data from different sources, like visual movements from cine MRI and speech acoustics.
• Predicting one modality from another is challenging, especially with limited data.

Purpose of the Study:

• To develop a novel synthesis framework for translating cine MRI sequences to spectrograms.
• To improve multimodal representation learning and understanding the relationship between speech and MRI data.

Main Methods:

• A fully convolutional heterogeneous translator with a 3D CNN encoder and 2D transpose convolution decoder.
• Latent space representation disentanglement using pairwise correlation of samples with the same speech word.
• Generative adversarial networks (GANs) for enhanced realism in generated spectrograms.

Main Results:

• The proposed framework demonstrated improved synthesis accuracy compared to existing methods.
• Successful translation from limited cine MRI sequences to spectrograms.
• Validation using 63 cine MRI sequences and corresponding speech acoustics.

Conclusions:

• The novel framework effectively translates cine MRI to spectrograms, even with limited data.
• The approach enhances the realism and accuracy of generated spectrograms.
• This work has potential applications in understanding the relationship between speech production and vocal tract movements.