Multimodal image and spectral feature learning for efficient analysis of water-suspended particles

We have developed a method to combine morphological and chemical information for the accurate identification of different particle types using optical measurement techniques that require no sample preparation. A combined holographic imaging and Raman spectroscopy setup is used to gather data from six different types of marine particles suspended in a large volume of seawater. Unsupervised feature learning is performed on the images and the spectral data using convolutional and single-layer autoencoders. The learned features are combined, where we demonstrate that non-linear dimensional reduction of the combined multimodal features can achieve a high clustering macro F1 score of 0.88, compared to a maximum of 0.61 when only image or spectral features are used. The method can be applied to long-term monitoring of particles in the ocean without the need for sample collection. In addition, it can be applied to data from different types of sensor measurements without significant modifications.