CellViT⁺⁺: Energy-efficient and adaptive cell segmentation and classification using foundation models.

Area of Science:

• Computational pathology
• Digital pathology
• Machine learning for medical imaging

Background:

• Deep learning models for cell segmentation require extensive annotated data and are computationally expensive.
• Existing methods lack adaptability to new cell types, creating bottlenecks in research and clinical workflows.
• CellViT++ is introduced to address these limitations in cell segmentation and classification.

Purpose of the Study:

• To develop a data-efficient and lightweight framework for generalized cell segmentation.
• To enable rapid adaptation of models to novel cell taxonomies with minimal data.
• To reduce computational costs and reliance on expert annotations in digital pathology.

Main Methods:

• CellViT++ utilizes a Vision Transformer with a frozen pretrained foundation model for segmentation.
• Deep cell embeddings are extracted during the forward pass at no extra computational cost.
• A lightweight classifier is trained on these embeddings for new cell type adaptation, and an automated workflow for training data generation from H&E and IF slides is demonstrated.

Main Results:

• CellViT++ achieves remarkable zero-shot segmentation and data efficiency, outperforming competing methods on public datasets.
• Superior results were obtained using only 10% of the training data on the CoNSeP dataset.
• The classifier approach drastically reduces training times (minutes vs. hours) and CO2 emissions by 96.93%, with models trained on automated labels showing comparable or superior performance to expert-annotated datasets.

Conclusions:

• CellViT++ is a robust, efficient, and open-source framework decoupling segmentation from classification in computational pathology.
• The framework's adaptability to new cell types with minimal data and automated dataset generation significantly reduce the need for expert annotation.
• CellViT++ serves as a foundational tool to accelerate research, enhance diagnostics, and enable deeper cohort analysis.