SpatialDINO: A Self-Supervised 3D Vision Transformer that enables Segmentation and Tracking in Crowded Cellular...

Area of Science:

• * Computational Biology
• * Bioimaging
• * Machine Learning for Microscopy

Background:

• * Analyzing complex cellular dynamics in 3D fluorescence microscopy is hindered by challenges in object identification, segmentation, and tracking within crowded, low-contrast environments.
• * Classical segmentation pipelines and supervised deep learning methods struggle with data variability, requiring extensive manual annotations that are costly and time-consuming.
• * Existing methods often fail to generalize across different imaging conditions, object shapes, and sizes, limiting broader application in quantitative cellular analysis.

Purpose of the Study:

• * To introduce SpatialDINO, a fully automated, self-supervised method for robust object detection, segmentation, and tracking in challenging 3D microscopy datasets.
• * To develop a native 3D vision transformer that learns dense volumetric representations without the need for voxel-level annotations.
• * To overcome the limitations of existing segmentation and tracking methods in heterogeneous, crowded cellular environments.

Main Methods:

• * SpatialDINO utilizes a modified DINOv2 architecture, training a native 3D vision transformer on unlabeled 3D fluorescence microscopy volumes.
• * The method learns semantic feature maps directly from single-channel, multi-channel, and anisotropic image data, accommodating varying z-spacings and imaging modalities.
• * It employs a self-supervised learning strategy, eliminating the requirement for manual voxel-level annotations or retraining for new datasets.

Main Results:

• * SpatialDINO successfully detects and segments objects of diverse sizes and shapes, including clathrin-coated pits, vesicles, endosomes, and lysosomes, in crowded cellular environments.
• * The method demonstrates generalization capabilities, enabling detection of previously unseen object classes like plasma membranes and nuclei, and even tumors in MRI scans.
• * SpatialDINO-derived features, combined with spatial proximity, significantly improve the tracking of endosomes in 4D time-series data, handling occlusion and appearance changes effectively.

Conclusions:

• * SpatialDINO provides a powerful, self-supervised foundation model for analyzing 3D fluorescence microscopy images, significantly lowering barriers to quantitative analysis.
• * The approach enables automated, accurate detection, segmentation, and tracking in complex, anisotropic 3D/4D datasets across different microscopy modalities without retraining.
• * By learning dense volumetric features directly from unlabeled data, SpatialDINO reduces reliance on manual annotation and enhances performance in challenging cellular imaging scenarios.