A universal medical imaging modality translation model in brain and head-and-neck radiotherapy

We developed a universal image translation model-Translate Any Modality (TAM) model to synthesize MRI sequences and CT for brain and head-and-neck radiotherapy, facilitating downstream clinical tasks including multimodal registration and treatment dose calculation.

We retrospectively curated multi-modal imaging from 90 patients (43 brain, and 47 head-and-neck), each with up to eight MRI sequences and a paired CT. TAM uses a two-stage translation strategy: a 3D U-Net to firstly segment soft tissues and bones to serve as anatomical anchors; and a conditional diffusion model, guided by these masks, to synthesize the target modality from any available input modalities. We evaluated 46 MRI-to-MRI and MRI-to-CT translation tasks against two comparison methods, CycleGAN and UNIT, using Frechet Inception Distance (FID), peak signal-to-noise-ratio (PSNR), structural similarity index (SSIM), and Hounsfield unit (HU)-based mean absolute error (MAE). Clinical relevance was tested via multi-modality registration with simulated B-spline deformations and dose calculation using clinical plans with Gamma criteria 1 %/1 mm and 2 %/2 mm.

TAM improved image quality (PSNR 30.0 ± 3.7; SSIM 0.973 ± 0.024) versus UNIT (27.0 ± 3.2; 0.958 ± 0.026) and CycleGAN (25.4 ± 4.1; 0.949 ± 0.037). For synthesized CT, TAM-based CT achieved average Gamma indices of 99.1 ± 1.7 % (2 %/2 mm) and 94.1 ± 6.4 % (1 %/1 mm) versus CycleGAN 93.1 ± 7.5 %/84.0 ± 9.5 % and UNIT 93.8 ± 7.0 %/86.3 ± 9.4 %, with dose-volume histograms closely matching real-CT plans.

TAM is a novel any-to-any medical image modality translation model. It provides flexible, anatomically faithful translation among MRI sequences and CT, which subsequently improves the accuracy of downstream tasks, including registration and dose calculation.