ViT-radiomics fusion for lymph node classification in ultrasound images: a multicenter study

Ultrasound (US) is widely used for assessing lymph node (LN) status, but its diagnostic accuracy remains highly operator dependent. A robust computer-assisted diagnostic model may enhance clinical performance and improve inter-operator and inter-center consistencies.

To develop and validate a multimodal fusion model, ViT-Rad, that combines radiomics features and deep learning features derived from vision transformers (ViT) for the classification of benign and malignant LNs in US images.

Between February 2016 and November 2023, a total of 1647 ultrasound images were retrospectively collected for analysis. In this multicenter study, we constructed ViT-Rad, a three-module neural network integrating ViT-based global contextual features and radiomics features extracted from manually delineated regions of interest. To address potential cross-center domain shift, we further employed weak/strong augmentation and a few-shot domain adaptation strategy using limited labeled external-center samples.

The model was trained and evaluated on a dataset from Center 1 (n = 1273; mean ± SD age, 57 ± 14 years), and its generalizability was tested on an external dataset from Center 2 (n = 374; mean ± SD age, 52 ± 18 years). ViT-Rad achieved an AUC of 0.95 [95% CI 0.91, 0.98] and an accuracy of 0.90 [95% CI 0.85, 0.95] on the internal test set, outperforming conventional radiomics models (AUC = 0.79, 95% CI 0.71, 0.89; P = .006). With domain adaptation, its AUC on the external set increased from 0.73 [95% CI 0.69, 0.79] to 0.85 [95% CI 0.81, 0.90]. These findings suggest improved adaptation-assisted external performance under cross-center domain shift.

By combining radiomics and ViT-derived features, ViT-Rad effectively integrates domain-specific and global contextual information, improving internal diagnostic performance and showing improved external adaptability after few-shot domain adaptation for LN classification on US images.