Investigation of Audio Controlled Edge Semantic Segmentation System for Visually Impaired People

This work presents an investigation into an audio-controlled edge semantic segmentation system motivated to enhance assistive technology for visually impaired people. Our multimodal system leverages audio inputs to actively control semantic segmentation processes. By utilizing separate convolutional networks for visual semantic segmentation and audio keyword spotting, the proposed system enables users to select specific object categories through simple voice commands. This selection dynamically filters visual information, reducing cognitive load and optimizing the display of essential environmental details. We provide an analysis of the trade-off between segmentation accuracy and computational efficiency across varying image resolutions and propose several post-processing techniques to enhance the user experience. The feasibility of running the system on edge is demonstrated with the Nvidia Jetson AGX Orin platform.