LoRA-INT8 Whisper: A Low-Cost Cantonese Speech Recognition Framework for Edge Devices

Area of Science:

• Speech Recognition
• Artificial Intelligence
• Machine Learning

Background:

• Low-resource automatic speech recognition (ASR) for Cantonese faces challenges with data scarcity, large models, and slow inference.
• Edge deployment of ASR systems requires efficient models that balance accuracy, speed, and size.

Purpose of the Study:

• To develop a cost-effective Cantonese ASR system for low-resource and edge-deployment settings.
• To address bottlenecks in data scarcity, model size, and inference speed.

Main Methods:

• Parameter-efficient fine-tuning of Whisper-tiny using LoRA (Low-Rank Adaptation) with rank = 8 on the Common Voice zh-HK dataset.
• INT8 quantization of the fine-tuned model using dynamic quantization in ONNX Runtime to create a compact checkpoint.
• Evaluation of the quantized model's performance on CPU and GPU, measuring character error rate (CER) and real-time factor (RTF).

Main Results:

• LoRA fine-tuning reduced CER from 49.5% to 11.1%, approaching full fine-tuning performance while significantly decreasing training costs.
• The INT8 quantized model achieved a 60 MB size, with RTF = 0.20 on CPU (5x real-time) and RTF = 0.06 on GPU, demonstrating substantial speed improvements.
• Ablation studies confirmed the LoRA-INT8 configuration provides an optimal trade-off between accuracy, speed, and model size.

Conclusions:

• The proposed LoRA-INT8 Cantonese ASR system offers a practical solution for resource-constrained environments.
• The system achieves significant improvements in efficiency and performance, paving the way for real-time ASR on edge devices.
• Future work aims to further enhance performance and energy efficiency for practical IoT applications.