Develop intelligent waste bin prototype based on fusion feature recognition of sounds and RGB images

Sorting municipal solid waste (MSW) at the source is a critical first step toward achieving a circular economy. Previous research has primarily focused on vision-based intelligent algorithms for MSW classification using red-green-blue (RGB) images. Some studies have also explored sound-based classification; however, only a few studies have employed multimodal deep learning methods (MDLMs) that integrate both image and sound data for MSW identification. To address this gap, the present study developed an innovative intelligent waste bin (IWB) prototype that utilizes MDLM and multimodal data fusion to enhance MSW classification. First, the sorting bin was constructed and an MSW dataset was constructed containing both audio signals and RGB images. The audio data were converted into Mel-frequency cepstral coefficients (MFCCs), and feature extraction from RGB images and MFCCs was performed using ResNet-101 and a Long Short-Term Memory network (LSTM) network, respectively. These features were then concatenated and fused within the MDLM model to classify MSW, which in turn controlled servomotors and rotatable boards in the IWB to automatically sort the waste. The results of this study clearly indicate that using sound modality alone is more effective for MSW identification than using RGB images. Based on fivefold cross-validation results, the MDLM model generally achieved higher average accuracy than the LSTM model, which used only sound features. However, the improvement was relatively modest, as the LSTM already achieved a high accuracy of approximately 90%. In the actual MSW sorting test using the IWB, the MDLM model also outperformed the LSTM, further confirming the advantages of multimodal fusion. Future research could explore the inclusion of a larger number of RGB images to develop more robust fusion features and further enhance the performance of the MDLM model.