A double-branch convolutional neural network model for species identification based on multi-modal data.

Area of Science:

• Spectroscopy
• Chemometrics
• Machine Learning

Background:

• Deep learning, particularly Convolutional Neural Networks (CNNs), is increasingly used for species identification via Raman spectra.
• CNNs struggle to fully extract features from overlapping or weak peaks in similar molecules, impacting identification accuracy.
• Multi-modal data fusion offers a more comprehensive analysis than single-modal data for complex samples.

Purpose of the Study:

• To develop a robust multi-modal deep learning model for enhanced species identification.
• To address the limitations of single-modal CNNs in extracting critical spectral features.
• To improve the accuracy and comprehensiveness of species identification in complex samples.

Main Methods:

• Proposed a novel double-branch CNN model (SI-DBNet) integrating Raman and image multi-modal data.
• Developed a 1D CNN for spectral branching incorporating dilated convolutions and efficient channel attention.
• Utilized Grad-CAM for visualizing model attention to key spectral regions.

Main Results:

• The SI-DBNet model achieved a superior classification accuracy of 98.8%.
• Demonstrated enhanced performance compared to single-modal and other multi-modal classification methods.
• Visualizations confirmed the model's focus on relevant spectral features.

Conclusions:

• The proposed SI-DBNet model effectively fuses Raman and image data for accurate species identification.
• Multi-modal data fusion significantly improves upon single-modal approaches for complex spectral analysis.
• This method provides a valuable new reference for species identification using integrated data.