Oral epithelial cell segmentation from fluorescent multichannel cytology images using deep learning

Cytology is a proven, minimally-invasive cancer screening and surveillance strategy. Given the high incidence of oral cancer globally, there is a need to develop a point-of-care, automated, cytology-based screening tool. Oral cytology image analysis has multiple challenges such as, presence of debris, blood cells, artefacts, and clustered cells, which necessitate a skilled expertise for single-cell detection of atypical cells for diagnosis. The main objective of this study is to develop a semantic segmentation model for Single Epithelial Cell (SEC) separation from fluorescent, multichannel, microscopic oral cytology images and classify the segmented images.

We have used multi-channel, fluorescent, microscopic images (number of images; n = 2730), which were stained differentially for cytoplasm and nucleus. The cytoplasmic and cell membrane markers used in the study were Mackia Amurensis Agglutinin (MAA; n: 2364) and Sambucus Nigra Agglutinin-1 (SNA-1; n: 366) with a nuclear stain DAPI. The cytology images were labelled for SECs, cluster of cells, artefacts, and blood cells. In this study, we used encoder-decoder models based on the well-established U-Net architecture, modified U-Net and ResNet-34 for multi-class segmentation. The experiments were performed with different class combinations of data to reduce imbalance. The derived MAA dataset (n: 14,706) of SEC, cluster, and artefacts/blood cells were used for developing a classification model. InceptionV3 model and a new custom Convolutional-Neural-Network (CNN) model (Artefact-Net) were trained to classify SNA-1 marker stained segmented images (n:6101). For segmentation models, Intersection Over Union (IoU) and F1 score were used as the evaluation matrices, while the classification models were evaluated using the conventional classification metrics like precision, recall and F1-Score.

The U-Net and the modified U-Net models gave the best IoU overall (0.73-0.76) as well as for SEC segmentation (079). The images segmented using the modified U-Net model were classified by Artefact-Net and Inception V3 model with F1 scores of 0.96 and 0.95 respectively. The Artefact-Net, when compared to InceptionV3, provided a better precision and F1 score in classifying clusters (Precision: 0.91 vs 0.80; F1: 0.91 vs 0.86).

This study establishes a pipeline for SEC segmentation with the segmented component containing only single cells. The pipline will enable automated, cytology-based early detection with reduced bias.