Interpretable polyp classification via end-to-end Concept Bottleneck Models with vision-language concept alignment

Artificial intelligence has improved adenoma detection during colonoscopy, but most models remain black boxes, limiting clinical trust, especially for sessile serrated lesions (SSLs). Concept Bottleneck Models (CBMs) route predictions through human-understandable concepts. We propose an end-to-end CBM for colonoscopic polyp classification with concept-level explanations aligned with clinical standards.

We defined 62 concepts from NBI International Colorectal Endoscopic (NICE), Japan NBI Expert Team (JNET), and Paris classification systems covering color, morphology, vascular/pit patterns, and contextual features. BiomedCLIP generated pseudo concept labels by patch-level image-text cosine similarity with max-pooling; these were not expert concept annotations. A concept projection layer with ReLU activation was inserted between an EfficientNet-B3 backbone and a linear classifier. Training used cross-entropy plus mean squared error concept alignment (lambda = 1.0). Evaluation used 508 images (192 SSL, 170 conventional, 146 malignant) and 5-fold image-level stratified cross-validation.

The CBM achieved macro F1 of 95.20% (±1.89%; 95% CI 92.58-97.82%), comparable to the black-box baseline of 94.97% (±2.52%; 95% CI 91.48-98.47%). Paired testing showed no significant fold-level difference (paired t-test, p = 0.835; Wilcoxon, p = 1.000), supporting comparability rather than superiority. Alignment strength showed an overall upward trend within the CBM family (F1 92.90% at lambda = 0 to 94.39% at lambda = 1.0). Concept ablation suggested distributed concept use, with no single concept causing more than a 0.80% F1 drop. Per-class F1 was 95.11% for SSL, 92.87% for conventional polyps, and 97.63% for malignant lesions.

Vision-language-aligned end-to-end CBM achieved interpretable polyp classification while maintaining black-box-level performance. Concept alignment may act as a regularization-like constraint, requiring further validation. This framework enables clinicians to inspect concept-level evidence for AI-assisted colonoscopy.