NucVerse3D: generalizable 3D nuclear instance segmentation across heterogeneous microscopy modalities

Accurate three-dimensional (3D) nuclear instance segmentation is a prerequisite for quantitative phenotyping in volumetric microscopy, yet remains challenging in densely packed tissues, irregular nuclear morphologies, and across heterogeneous imaging modalities. Here we present NucVerse3D, a deep-learning framework for generalized 3D nuclei instance segmentation that combines a residual attention 3D U-Net architecture with a reversible gradient-field representation for robust centroid-aware instance reconstruction. NucVerse3D is trained end to end in 3D using modality-agnostic preprocessing and isotropic scale normalization, enabling deployment across confocal microscopy, two-photon microscopy, light-sheet microscopy, micro-computed tomography, and scanning electron microscopy volumes. We benchmarked NucVerse3D on seven volumetric datasets spanning multiple species and tissues, comprising more than forty thousand manually annotated nuclei, including newly released ground-truth datasets of mouse liver tissue (control and hepatocellular carcinoma) and Drosophila brain glial nuclei. Across datasets, NucVerse3D achieved consistently high precision and competitive recall, resulting in strong F1-scores and average precision across a wide range of imaging conditions. While a modest precision-recall imbalance is observed in certain datasets, favoring high-confidence detections, this behavior reflects a conservative instance reconstruction strategy that prioritizes accurate boundary delineation and reduces false positive segmentation in densely packed and morphologically heterogeneous tissues. A single generalized model trained on pooled data matched the performance of dataset-specific models, and ablation experiments demonstrated that preprocessing and scale normalization substantially contribute to performance under strict intersection-over-union criteria. To demonstrate the biomedical utility of NucVerse3D, we applied it to 3D liver images from a mouse model of hepatocellular carcinoma (HCC) to enable spatially resolved 3D nuclear phenotyping. In healthy liver tissue, nuclear DNA content and nuclear volume exhibited a tightly regulated log-log scaling relationship. In contrast, tumor-adjacent and tumor regions displayed progressive disruption of this coupling, forming spatially coherent domains of nuclear DNA-volume decoupling that are not detectable in conventional two-dimensional histology. We quantify this phenomenon using a Nuclear Decoupling Score (NDS), revealing increased nuclear instability aligned with pathological tissue remodeling highlighting NDS as a potential quantitative biomarker of dysplastic and tumor tissue. Together, NucVerse3D provides a robust and generalizable solution for 3D nuclear instance segmentation and enables quantitative nuclear phenotyping across imaging modalities.