Representation geometry and task performance in vision-language modeling for computed tomography enterography

Computed tomography (CT) enterography is central to inflammatory bowel disease evaluation, but optimal representation choices for vision-language learning on this volumetric modality are unclear. We establish baseline performance and quantify how aggregation, slice encoding, and retrieval context affect classification, retrieval, and report generation.

We fine-tuned vision-language models on abdominal CT enterography with parameter-efficient adapters and compared (i) mean versus attention pooling of slice embeddings, (ii) multiwindow red-green-blue Hounsfield-unit encoding versus multiplanar sampling for added spatial coverage, and (iii) report generation with versus without retrieval-augmented generation (RAG). A three-teacher pseudolabel framework enabled evaluation without expert annotations. Classification and retrieval comparisons used three training seeds.

Mean and attention pooling achieved the same mean three-class disease classification accuracy (55.7%; standard deviations of 2.0 and 2.4 percentage points) at the largest low-rank adaptation capacity. Neither consistently outperformed the other at matched capacity with central sampling. Sixteen slices at 20% to 80% had higher classification accuracy than 8 slices, 32 slices, and 10% to 90% sampling in all three seeds for both pooling methods. The tested multiplanar configurations had lower classification accuracy. For report generation, fine-tuning without retrieval performed at prevalence-matched chance for within-1 report-label agreement (70.4% versus 71.0% random). RAG improved within-1 report-label agreement by 7 to 14 points over chance and reduced ordinal mean absolute error from 0.98 to 0.80-0.89.

Slice sampling and tissue-contrast-focused encoding affect performance in CT enterography vision-language systems, and RAG improved report-label agreement over the fine-tuned baseline. These results provide initial baselines and practical guidance for multimodal learning in volumetric CT.