Deep Learning in Cytopathology: The Potential for Multimodal Synergy of Clinical and Cytological Input Data

The art of cytopathology lies not only in the interpretation of the slide but also careful consideration of demographic, clinical, radiological, and other relevant information of individual patients. While deep image learning may perform well in specific cytology image sets or uncover novel, significant cytomorphologic parameters, the lack of clinical context puts prediction models at a dangerous position where irrelevant and potentially harmful diagnoses may be issued. The increment in prediction accuracy by expanding the size of image sets diminishes and plateaus after reaching certain thresholds. This can be overcome by introducing multimodal data to deep learning, which has seen success in resection and small biopsy specimens from histopathological specimens and early adoption in cytopathology. In this study, a qualitative review on the current state of the potential sources of multimodal input-demographics, clinical investigations, and radiology; the diagnostic data that can be obtained in a cytology specimen-stains and preparations, immunocytochemistry, and molecular testing will be discussed with reference to the potential of multimodal deep learning in cytopathology.