CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Signal Processing

Background:

• Convolutional Neural Networks (CNNs) have dominated audio classification.
• Audio Spectrogram Transformers (ASTs) recently emerged, outperforming CNNs.
• Knowledge Distillation (KD) is a technique for model training.

Purpose of the Study:

• Investigate the interaction between CNNs and ASTs.
• Apply cross-model knowledge distillation (CMKD) to audio classification.
• Achieve state-of-the-art results using the proposed CMKD method.

Main Methods:

• Utilized CNNs and ASTs as both teachers and students.
• Implemented knowledge distillation (KD) for inter-model knowledge transfer.
• Evaluated the CNN/Transformer Cross-Model Knowledge Distillation (CMKD) method on benchmark datasets.

Main Results:

• Student models trained via KD significantly improved performance.
• In many cases, the student model outperformed its teacher model.
• New state-of-the-art results were achieved on FSD50K, AudioSet, and ESC-50 datasets.

Conclusions:

• CNNs and ASTs exhibit a complementary relationship in audio classification.
• Cross-model knowledge distillation (CMKD) effectively enhances model performance.
• The proposed CMKD method offers a promising direction for future audio classification research.