Real-time intraoperative diagnosis by deep neural network driven multiphoton virtual histology

Recent advances in label-free virtual histology promise a new era for real-time molecular diagnosis in the operating room and during biopsy procedures. To take full advantage of the rich, multidimensional information provided by these technologies, reproducible and reliable computational tools that could facilitate the diagnosis are in great demand. In this study, we developed a deep-learning-based framework to recognize cancer versus normal human breast tissue from real-time label-free virtual histology images, with a tile-level AUC (area under receiver operating curve) of 95% and slide-level AUC of 100% on unseen samples. Furthermore, models trained on a high-quality laboratory-generated dataset can generalize to independent datasets acquired from a portable intraoperative version of the imaging technology with a physics-based adapted design. Classification activation maps and final feature visualization revealed discriminative patterns, such as tumor cells and tumor-associated vesicles, that are highly associated with cancer status. These results demonstrate that through the combination of real-time virtual histopathology and a deep-learning framework, accurate real-time diagnosis could be achieved in point-of-procedure clinical applications.