Creating musical features using multi-faceted, multi-task encoders based on transformers.

Area of Science:

• Computational Musicology
• Machine Learning for Music
• Music Information Retrieval

Background:

• Computational intelligence drives music technologies for creation, sharing, and interaction.
• Supervised learning for music tasks requires extensive data and offers limited insights.
• Existing methods struggle with generalized computational music understanding.

Purpose of the Study:

• To develop a novel model for generating audio-musical features for enhanced music understanding.
• To leverage self-supervision and cross-domain learning for robust music representation.
• To improve performance on downstream music information retrieval tasks.

Main Methods:

• Pre-training a multi-faceted, multi-task music transformer (M3BERT) using masked reconstruction with self-attention bidirectional transformers.
• Fine-tuning output representations on various downstream music understanding tasks.
• Utilizing self-supervised and cross-domain learning strategies.

Main Results:

• M3BERT-generated features outperform other audio and music embeddings on diverse music-related tasks.
• Demonstrated effectiveness of self-supervised and semi-supervised learning for music modeling.
• Achieved robust performance across music genre detection and emotion recognition.

Conclusions:

• Self-supervised and semi-supervised learning offer a generalized and robust approach to computational music modeling.
• M3BERT provides a strong starting point for various music-related tasks.
• The model has potential applications in deep representation learning and robust music technology.