PyCLM: programming-free, closed-loop microscopy for real-time measurement, segmentation, and optogenetic stimulation

In cell biology, optical techniques are increasingly used to measure cells' internal states (biosensors) and to stimulate cellular responses (optogenetics). Yet the design of all-optical experiments is often manual: a pre-determined stimulus pattern is applied to cells, biosensors are measured over time, and the resulting data is processed off-line. With the advent of machine learning for segmentation and tracking, it becomes possible to envision closed-loop experiments where real-time information about cells' positions and states are used to dynamically determine optogenetic stimuli to alter or control their behavior. Here, we develop PyCLM, a Python-based suite of tools to enable real-time measurement, image segmentation, and optogenetic control of thousands of cells per experiment. PyCLM is designed to be as simple for the end user as possible, and multipoint experiments can be set up that combine a wide variety of imaging, image processing, and stimulation modalities without any programming. We showcase PyCLM on diverse applications: studying the effect of epidermal growth factor receptor activity waves on epithelial tissue movement, simultaneously stimulating ~1,000 single cells to guide tissue flows, and performing real-time feedback control of cell-to-cell fluorescence heterogeneity. This tool will enable the next generation of dynamic experiments to probe cell and tissue properties, and provides a first step toward precise control of cell states at the tissue scale.