Deep learning neural network of adenocarcinoma detection in effusion cytology

Cytologic examination, which confirms the presence or absence of malignant cells, detects malignant cells from various organs, with adenocarcinoma as the most common histologic type. We developed a deep learning model to detect malignant cells in images obtained following effusion cytology.

The deep learning model was created using the YOLOv8 object detection algorithm (Roboflow, Inc) and 275 cases of adenocarcinoma comprising 12 182 images and 29 245 labels as well as 188 cases negative for malignancy comprising 1980 images.

The adenocarcinoma test dataset exhibited Precision, Recall, F1, and mean average Precision scores of 0.909, 0.911, 0.910, and 0.955, respectively. The number of adenocarcinoma test images in which 1 or more malignant cells were detected was 2710 of 2731. The sensitivity in the nonadenocarcinoma dataset was 97.1%, and the false-positive rate in the negative-for-malignancy dataset was 7.3%. The accuracy, sensitivity, and specificity of the model using all the test datasets were 96.3%, 98.5%, and 92.7%, respectively.

Although some issues regarding cell annotation when creating an object detection model remain, the accuracy is sufficient to assist cancer screening in effusion cytology. It is vital to reliably detect malignant cells in effusion cytology, and the further development of automated systems to reduce false-negative results is expected.