Recent advances in deep learning for biological microscopy image analysis beyond segmentation

In the past decade, we have witnessed an unprecedented growth of artificial intelligence (AI) techniques for high-resolution microscopy images. More importantly, we are observing a paradigm evolution of these deep learning (DL) methods, from a post-hoc analysis tool to nowadays an essential component in imaging-based biological researches. Building on the foundational success of DL in segmentation, we hope to elucidate in this review how AI is making an era of automated biological discovery embedded throughout the research cycle beyond segmentation. We structure our discussion along different research stages, from assay development, image acquisition, image and data analysis, to biological modeling and interpretation. First, in assay design and image acquisition, we illustrate how integrating DL-based computational strategies at the experimental stage enables efficient and information-rich assay designs, which may not even be possible before in conventional settings. Next, we examine image analysis, highlighting the transition from handcrafted features to the automated discovery of complex phenotypes and dynamic behaviors. Subsequently, we explore interpretation and modeling, discussing how AI facilitates extraction of biological insights from quantitative data. At the end, we discuss the importance of and current efforts in model evaluation and validation, and the rising of closed-loop microscopy concept.