Sample-level sound synthesis with recurrent neural networks and conceptors

Area of Science:

• Artificial Intelligence
• Computer Music
• Signal Processing

Background:

• Conceptors are a recent development in reservoir computing, influencing recurrent neural networks (RNNs) to generate arbitrary patterns.
• They enable pattern interpolation, extrapolation, and boolean logic for pattern combination.
• The application of conceptors to sound synthesis, particularly for multi-timbre audio and scalability, remains largely unexplored.

Purpose of the Study:

• To introduce and evaluate a novel sound synthesis method, Conceptular Synthesis, based on conceptors.
• To explore the potential of conceptors for generating and manipulating multi-timbre audio patterns.
• To assess the quality, malleability, and limitations of conceptor-based sound synthesis compared to existing methods.

Main Methods:

• Developed Conceptular Synthesis, a granular synthesis-based method utilizing sets of conceptors trained on a single RNN.
• Implemented a runtime mechanism to switch between conceptors, generating and recombining short patterns into longer audio signals.
• Experimentally evaluated sound resynthesis quality and compared malleability with echo state network models.

Main Results:

• Conceptor models achieved comparable audio resynthesis quality to echo state networks with stored patterns.
• Conceptor models demonstrated superior malleability and potential for creative sound manipulation.
• Limitations identified include reproduction quality issues and computational/memory constraints for longer audio samples.

Conclusions:

• Conceptular Synthesis shows significant potential as a novel sound synthesis technique for computer music and sound design.
• Conceptors offer unique creative possibilities through pattern morphing, boolean logic, and RNN dynamic manipulation.
• Further research is needed to address limitations and explore broader applications of conceptors in audio generation.