Multimodal image and spectral feature learning for efficient analysis of water-suspended particles

Area of Science:

• Environmental science
• Analytical chemistry
• Optical physics

Background:

• Accurate identification of marine particles is crucial for understanding ocean processes.
• Current methods often require sample preparation, limiting real-time analysis.
• Multimodal data fusion offers potential for enhanced particle characterization.

Purpose of the Study:

• To develop a sample-free method for identifying marine particle types.
• To combine morphological and chemical information for improved particle classification.
• To enable automated, long-term monitoring of oceanic particles.

Main Methods:

• Utilized a combined holographic imaging and Raman spectroscopy system.
• Applied unsupervised feature learning (convolutional and single-layer autoencoders) to image and spectral data.
• Employed non-linear dimensionality reduction on combined multimodal features.

Main Results:

• Achieved a high clustering macro F1 score of 0.88 using combined features.
• Demonstrated superior performance compared to using only image (max F1 0.61) or spectral data.
• Successfully classified six different types of marine particles.

Conclusions:

• The developed multimodal approach significantly enhances marine particle identification accuracy.
• This method eliminates the need for sample collection, facilitating in-situ, long-term ocean monitoring.
• The technique is adaptable to various sensor measurements with minimal modifications.