Cascaded Deep Learning-Based Model for Classification and Segmentation of Plaques from Carotid Ultrasound Images

Carotid plaque classification based on ultrasound echogenicity and quantification of plaque burden are crucial in stroke risk assessment. In this work, we propose a framework that leverages the synergy between classification and segmentation by sharing plaque location information to enhance the performance of both tasks. Our cascaded framework integrates a ResNet-based classifier (Masked-ResNet-DS) with MedSAM, a medically adapted version of the Segment Anything Model for joint classification and segmentation of carotid plaques from 2D ultrasound images. Ground truth boundaries are used to guide region-specific feature pooling in the classifier, helping it focus on plaques during training. Since ground truth boundaries are unavailable at inference, we introduce a two-iteration strategy: the first generates a class activation map (CAM), which is then used for focused pooling in the second iteration to predict plaque type. The CAM is also used as a prompt to guide MedSAM for segmentation. To ensure accurate localization, the CAM is supervised during training using a Dice loss against the segmentation ground truth. Masked-ResNet-DS achieves a mean F1-score of 96.7% in plaque classification, at least 3.2% higher than competing methods. Ablation studies confirm that ground truth-based pooling and CAM supervision both improve classification. CAM-guided MedSAM achieves a Dice similarity coefficient (DSC) of 86.6%, outperforming U-Net and nnU-Net by 5.9% and 3.6%, respectively. In addition, CAM prompts improve MedSAM's DSC by 2.2%. By sharing plaque location between classification and segmentation, the proposed method improves both tasks and provides a more accurate tool for stroke risk stratification.