The Challenge of External Generalisability: Insights from the Bicentric Validation of a [⁶⁸Ga]Ga-PSMA-11 PET Based...

Background: PSMA PET radiomics is a promising tool for primary prostate cancer (PCa) characterisation. However, small single-centre studies and lack of external validation hinder definitive conclusions on the potential of PSMA PET radiomics in the initial workup of PCa. We aimed to validate a radiomics signature in a larger internal cohort and in an external cohort from a separate centre. Methods: One hundred and twenty-seven PCa patients were retrospectively enrolled across two independent hospitals. The first centre (IRCCS San Raffaele Scientific Institute, Centre 1) contributed 62 [⁶⁸Ga]Ga-PSMA-11 PET scans, 20 patients classified as low-grade (ISUP grade < 4), and 42 as high-grade (ISUP grade ≥ 4). The second centre (Stanford University Hospital, Centre 2) provided 65 [⁶⁸Ga]Ga-PSMA-11 PET scans, and 49 low-grade and 16 high-grade patients. A radiomics model previously generated in Centre 1 was tested on the two cohorts separately and afterward on the entire dataset. Then, we evaluated whether the radiomics features selected in the previous investigation could generalise to new data. Several machine learning (ML) models underwent training and testing using 100-fold Monte Carlo cross-validation, independently at both Centre 1 and Centre 2, with a 70-30% train-test split. Additionally, models were trained in one centre and tested in the other, and vice versa. Furthermore, data from both centres were combined for training and testing using Monte Carlo cross-validation. Finally, a new radiomics signature built on this bicentric dataset was proposed. Several performance metrics were computed. Results: The previously generated radiomics signature resulted in an area under the receiver operating characteristic curve (AUC) of 80.4% when tested on Centre 1, while it generalised poorly to Centre 2, where it reached an AUC of 62.7%. When the whole cohort was considered, AUC was 72.5%. Similarly, new ML models trained on the previously selected features yielded, at best, an AUC of 80.9% for Centre 1 and performed at chance for Centre 2 (AUC of 49.3%). A new signature built on this bicentric dataset reached, at best, an average AUC of 91.4% in the test set. Conclusions: The satisfying performance of radiomics models when used in the original development settings, paired with the poor performance otherwise observed, emphasises the need to consider centre-specific factors and dataset characteristics when developing radiomics models. Combining radiomics datasets is a viable strategy to reduce such centre-specific biases, but external validation is still needed.