Musical timbre style transfer with diffusion model.

Area of Science:

• Audio Signal Processing
• Machine Learning
• Digital Music

Background:

• Timbre transfer aims to change an audio sample's instrument characteristics while preserving pitch and melody.
• Existing methods, often using image-to-image style transfer, produce unsatisfactory results with waveform artifacts.
• Diffusion models excel in high-quality image generation, offering a promising avenue for audio synthesis.

Purpose of the Study:

• To develop an advanced timbre transfer technique using diffusion models.
• To overcome limitations of current models that generate audio with unrelated waveforms.
• To achieve high-fidelity timbre transfer while maintaining original audio properties.

Main Methods:

• Audio waveform converted to Constant-Q Transform (CQT) spectrogram.
• Image-to-image conversion techniques applied to CQT spectrograms for timbre transfer.
• DiffWave model used to reconstruct the modified CQT spectrogram back into an audio waveform.

Main Results:

• The proposed diffusion-based model demonstrated superior performance in both one-to-one and many-to-many timbre transfer tasks.
• Experimental results showed significant improvements compared to baseline timbre transfer models.
• Generated audio samples exhibited better quality and preservation of musical elements.

Conclusions:

• The diffusion model offers a promising approach for high-quality audio timbre transfer.
• This technique advances the state-of-the-art in musical audio manipulation.
• The method effectively addresses challenges in preserving audio fidelity during timbre transformation.