Plastic waste identification based on multimodal feature selection and cross-modal Swin Transformer

The classification and recycling of municipal solid waste (MSW) are strategies for resource conservation and pollution prevention, with plastic waste identification being an essential component of waste sorting. Multimodal detection of solid waste has increasingly replaced single-modal methods constrained by limited informational capacity. However, existing hyperspectral feature selection algorithms and multimodal identification methods have yet to leverage cross-modal information exhaustively. Therefore, two RGB-hyperspectral image (RGB-HSI) multimodal instance segmentation datasets were constructed to support research in plastic waste sorting. A feature band selection algorithm based on the Activation Weight function was proposed to automatically select influential hyperspectral bands from multimodal data, thereby reducing the burden of data acquisition, transmission, and inference. Furthermore, the multimodal Selective Feature Network (SFNet) was introduced to balance information across various modalities and stages. Moreover, the Correlation Swin Transformer Block was proposed, specifically crafted to fuse cross-modal mutual information, which can be synergistically employed with SFNet to enhance multimodal recognition capabilities further. Experimental results show that the Activation Weight band selection function can select the most effective feature bands. At the same time, the Correlation SF-Swin Transformer achieved the highest F1-scores of 97.85% and 97.37% in the two plastic waste object detection experiments, respectively. The source code and final models are available at https://github.com/Bazenr/Correlation-SFSwin, and the dataset can be accessed at https://www.kaggle.com/datasets/bazenr/rgb-hsi-rgb-nir-municipal-solid-waste.