End-to-End Segmentation and Classification of Zooplankton Using Shadowgraphy and Convolutional Neural Networks

With in situ imaging systems becoming more common, precise, and economically viable, use of these systems has grown dramatically, including both automated classification and biomass estimations. However, a rather large and overlooked portion of these efforts is reliable detection and classification of these organisms as they pass through the imaging device. This paper focuses on the development of an end-to-end classification CNN-based algorithm for marine zooplankton using the in situ Ichthyoplankton Imaging System (ISIIS-DPI) from Bellamare (La Jolla, CA, USA). Our novel approach considers many issues with automated segmentation and classification, including over-segmentation, noise segmentation, and organism size input. This allows for classifications in diverse water types, demonstrated by the comparison of three datasets created in conjunction with this project, each with very different water properties and zooplankton communities (Florida Gulf coast; Trondheimsfjord, Norway; Sargasso Sea). Our segmented image dataset contains 70,624 regions of interest (ROIs) across four organism classes-Chaetognath, Crustacean, Gelatinous, and Larvacean-with two classes dedicated to detritus. Four common network architectures-Resnet, Xception, GoogleNet, and Darknet-are trained on this dataset, with final test accuracies in the range of 95.94-96.09%. Following this initial training, a secondary level of classification is introduced. The base Gelatinous class is further divided into six groups. The same four CNN architectures are used once again, with final accuracies in the range of 86.12-90.40%, showing the ability to taxonomically classify down to the order level. The present work introduces a versatile, adaptable, scalable and autonomous segmentation and classification algorithm using niched networks mirroring taxonomy, and is fully contained in a publicly available MATLAB R2025a custom graphical user interface.