Foundation model cascades enable zero-shot microscopy image analysis for cell therapy manufacturing

The scalable manufacturing of cell therapies creates a significant need for robust process analytical technologies, where automated analysis of noninvasive microscopy images offers a powerful method for monitoring critical quality attributes. However, conventional machine-learning models are often bottlenecked by extensive data labeling and poor generalizability across different batch effects. To overcome these limitations, we introduce a foundational model cascade for the zero-shot analysis of microscopy images.

In the first stage, a multimodal large language model (LLM) detects anomalies, and anomalous images immediately trigger an alert. Otherwise, the segment anything model performs exhaustive instance segmentation, and the detected objects are classified by the LLM to estimate cell counts and viability.

This unified, zero-shot approach delivers robust anomaly detection together with quantitative measures of cell count and health, without any task-specific fine-tuning.

By combining pre-trained foundation models in a complementary cascade, our method provides a generalizable solution for real-time process monitoring and feedback control, paving the way for more scalable and automated cell therapy manufacturing.