Learned Image Coding With Generative Reference of Conditional Latents

This paper presents a learned image compression framework that exploits semantically correlated external images as conditional coding references in the latent domain. Three reference generation methods are investigated: local dictionary retrieval, web-based image search, and diffusion-based image-text-image generation. Based on rate-distortion optimization in the latent domain, we learn a network to synthesize a conditional latent map from these correlated images, providing an effective semantic reference for conditional coding of the input image. These generation processes are synchronized between the encoder and decoder with a very low communication overhead. Our theoretical analysis shows that our approach is robust to perturbations in reference samples, with a subspace recovery error bound that is inversely proportional to the correlation between the reference and original images. This indicates a high tolerance to matching errors of the reference images. Comprehensive experiments on benchmark datasets show that our proposed generative references of conditional latents (GRCL) method achieves significant improvements in coding performance, up to 1.5 dB PSNR gain over state-of-the-art methods with a very small bit-rate overhead, approximately 0.005 bits per pixel. Our work presents a new generic framework to utilize external references for learned image compression.