DeepSpectrumLite: A Power-Efficient Transfer Learning Framework for Embedded Speech and Audio Processing From...

Area of Science:

• Artificial Intelligence
• Machine Learning
• Signal Processing

Background:

• Deep neural networks for speech and audio processing are computationally intensive, hindering integration into embedded systems.
• Current constraints necessitate large datasets, significant computational power, and complex architectures, limiting real-world, real-time applications on devices.

Purpose of the Study:

• To introduce DeepSpectrumLite, an open-source, lightweight transfer learning framework for on-device speech and audio recognition.
• To overcome the limitations of traditional deep learning models in embedded systems by leveraging pre-trained Convolutional Neural Networks (CNNs).

Main Methods:

• Utilizing pre-trained image CNNs for audio classification through transfer learning.
• On-the-fly creation and augmentation of Mel spectrograms from raw audio signals.
• Finetuning CNNs for specific audio classification tasks and real-time deployment on mobile devices.

Main Results:

• Achieved real-time performance with a mean inference lag of 242.0 ms on a consumer-grade smartphone using DenseNet121.
• Demonstrated state-of-the-art results across various paralinguistic and general audio tasks, including emotion recognition and COVID-19 analysis.
• Operated decentralized, eliminating the need for data upload and enhancing user privacy.

Conclusions:

• DeepSpectrumLite offers a viable solution for efficient, on-device audio processing using transfer learning.
• The framework's lightweight nature and real-time capabilities make it suitable for diverse embedded applications.
• Open-source availability and comprehensive documentation facilitate adoption by researchers and developers.