Visual prompt tuning for task-flexible medical image synthesis

Medical image synthesis has broad applications in modality-to-modality translation, denoising, and super-resolution, when specific modalities are missing, various types of noise occur, or resolution discrepancies exist across modalities. The traditional approach requires a separate model for each task, making it inefficient, and nearly impossible to accommodate various tasks in medical image synthesis.

We introduce a task-agnostic medical image-synthesis model utilizing prompt tuning that leverages a diffusion model and prompt tuning to fine-tune large capacity pretrained models efficiently. Our method can handle multiple tasks that cover various input-output combinations in a single model.

Our method can perform denoising, translation, super-resolution, and tumor inpainting tasks for brain MRI and abdominal CT. Through quantitative and qualitative evaluations, we demonstrate that our model achieves the best performance in terms of FID scores across all evaluated tasks. Our multi-task model achieves a PSNR of 25.76 and an SSIM of 0.908 for T1-to-T2 translation; a PSNR of 30.30 and an SSIM of 0.932 for denoising; a PSNR of 29.24 and an SSIM of 0.874 for super-resolution; and an FID of 16.18 with an LPIPS of 0.090 for tumor inpainting.

We proposed a method that enables task-agnostic medical image synthesis, allowing for the specification of the desired synthesis task, modality, and organ of the target image via prompt tuning. Our method can be extended to other modalities and organs. The code is available at https://github.com/jongdory/VPT-Med.