Developing Accessible Assistive Technology for the Deaf and Hard of Hearing by Deploying a Fine-Tuned Deep Neural...

This paper details the development of a deep neural network, and an accompanying application, capable of classifying and detecting five target classes important to the deaf and hard-of-hearing: a running faucet, a dripping faucet, a car engine, a car horn, and a fridge alarm. These classes were based on questionnaire results from previous research, which concluded with a long short-term memory (LSTM) model and an augmented self-captured dataset, but fell short of developing a mobile application as originally planned. To make such an application possible, this subsequent research primarily focused on the implementation of a "negative" class to account for the irrelevant sounds, those not part of the target classes, expected in real-world scenarios. The research therefore improved upon the previous results by designing a more capable model, while subsequent development served to embed this model into an application accessible from most mobile phones.The final model achieved an AUC of 0.97 ± 0.01, with a very well-balanced precision and recall, compromising between the conflicting tolerances fit for the critical and benign target classes. This was achieved through a novel approach for fine-tuning the YamNet audio classification model, where its scores - representing the ontology of the AudioSet dataset - were used as query for its embeddings, and an LSTM was used to flatten the resulting frames while retaining temporal information outside the scope of a single frame. To keep the performance overhead minimal, the scores and embeddings were resized using convolutional layers, resulting in a model fit for real-time use on mobile devices. To this end, the model was deployed to the Web using TensorFlow.js, and made available for offline use as a Progressive Web App. However, further research should focus on user testing to ensure the application and underlying model fulfil the needs of the target audience.