Advancing radiograph representation learning via cascading graph alignment for vision-language clinical concepts

Vision-language pre-training on paired radiology images and reports has shown great potential to promote downstream image tasks. However, its potential is far from fully unleashed when conducting implicit alignment for clinically significant concepts (e.g., anatomic and pathology findings) between visual and textual embeddings, as most prior works did. Without explicit exploration, it easily confuses critical medical concepts with non-essential background content, and fails to compactly investigate each image-report pair from a well-structured systematic point of view, causing miscalibrated semantic alignment across modalities. To address these issues, we proposed a Cascading Graph Alignment (CGA) framework to enable explicit alignment for the clinically significant concepts as well as the diagnostic content summary, to facilitate in-depth exploration of medically pertinent visual and textual representations during model pre-training. We started by identifying each type of clinical concept, including anatomic structures, pathology findings, and the relation between them in each image-report pair, and then structured them into a pair of isomorphic graphs for each modality to succinctly summarize its essential content. Here, we took the anatomic and pathological findings as the graph nodes and their relations as graph edges. We then employed a cascading alignment strategy to sequentially ensure cross-modal semantic consistency from anatomy nodes, pathology nodes, and relation edges toward the graph-structured content summary, progressively refining the investigation of clinically pertinent embeddings across modalities. We have extensively evaluated our framework on diverse downstream image tasks, including image classification, image segmentation, and object detection. Our model significantly outperformed the state-of-the-art methods by a great margin when fine-tuning in various annotation-efficient scenarios.