Related Experiment Video
Updated: May 3, 2026
Implantation and Monitoring by PET/CT of an Orthotopic Model of Human Pleural Mesothelioma in Athymic Mice
Published on: December 21, 2019
Mask-aware foundational-model embeddings for 18F-FDG-PET/CT prognosis in multiple myeloma
Javier Guinea-Pérez1, Silvia Uribe1, Sara Peluso2
1Universidad Politécnica de Madrid, Avenida Complutense, 30, Madrid, 28040, Madrid, Spain.
Purpose:
To test whether internal memory states from a medical foundational segmentation model can serve as compact, mask-aware embeddings for predicting progression-free survival (PFS) in multiple myeloma (MM) from whole-body [18F]FDG PET/CT, and how late fusion of PET, CT, and clinical data enhances prognostic performance.
Methods:
We analyzed 227 newly diagnosed MM patients with PET/CT and clinical data. For two regions of interest (spine-dilated and full skeleton), we prompted MedSAM2 slice-wise using mask-derived bounding boxes and cached the final spatio-temporal memory tensor per modality. We compared two downsampling strategy to obtain per-study embeddings: channel×memory averaging with a small CNN head, and depth-attention pooling. PET and CT embeddings were combined by late fusion and passed to a DeepSurv head. We evaluated image-only and multimodal (image+clinical) models with stratified 5-fold cross-validation. The primary endpoint was Harrell's c-index (mean ± SE across folds).
Results:
Image-only models using the averaging downsampler achieved up to 0.659±0.015 c-index (PET, spine-dilated), comparable to baseline radiomics results. Multimodal models improved discrimination to 0.710±0.032 (CT, spine-dilated), with similar performance for other PET/CT+clinical variants (0.703-0.710), improving clinical-only baselines ∼6.5%. Averaging consistently outperformed depth-attention; concatenation and gated fusion performed comparably. PET outperformed CT within the same mask in image-only settings.
Conclusion:
Mask-aware memory embeddings extracted from a foundational segmentation model provide effective, data-efficient imaging biomarkers for MM PFS and, when fused with routine clinical covariates, significantly improve risk stratification over clinical-only or radiomics baselines. This offers a practical path to prognostic modeling on small medical cohorts without feature design.
Insights
Internal memory states from segmentation models can predict multiple myeloma progression-free survival using PET/CT scans. Fusing imaging and clinical data significantly improves prognostic accuracy over existing methods.
Area of Science:
- Medical imaging analysis
- Artificial intelligence in oncology
- Radiomics and deep learning
Background:
- Predicting progression-free survival (PFS) in multiple myeloma (MM) is crucial for treatment planning.
- Current prognostic models often rely on clinical data or radiomics, with potential for improvement using advanced imaging features.
- Foundational segmentation models offer novel ways to extract information from medical images.
Purpose of the Study:
- To evaluate the efficacy of internal memory states from a medical segmentation model (MedSAM2) as compact, mask-aware embeddings for MM PFS prediction.
- To assess the impact of late fusion of PET, CT, and clinical data on prognostic performance.
- To determine if these embeddings can serve as data-efficient imaging biomarkers.
Main Methods:
- Analysis of 227 newly diagnosed MM patients with whole-body [18F]FDG PET/CT and clinical data.
- Extraction of spatio-temporal memory tensors from MedSAM2 using mask-derived bounding boxes for spine-dilated and full skeleton regions.
- Comparison of channel×memory averaging and depth-attention pooling for per-study embedding generation, followed by late fusion with clinical data and evaluation using DeepSurv.
Main Results:
- Image-only models using averaging achieved a c-index of 0.659±0.015 (PET, spine-dilated), comparable to radiomics.
- Multimodal models (PET/CT + clinical data) improved discrimination to 0.710±0.032 (CT, spine-dilated), outperforming clinical-only baselines by ~6.5%.
- Averaging downsampling strategy consistently outperformed depth-attention, and PET embeddings showed better performance than CT in image-only settings.
Conclusions:
- Mask-aware memory embeddings from foundational segmentation models are effective imaging biomarkers for MM PFS prediction.
- Fusion with clinical data significantly enhances risk stratification compared to clinical-only or radiomics approaches.
- This method provides a practical, data-efficient strategy for prognostic modeling in small medical cohorts without manual feature engineering.
