Instruction-Driven Multi-Weather Image Translation Based on a Large-Scale Image Editing Model

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Weather image translation is crucial for data augmentation but faces challenges with costly sample collection.
• Existing Generative Adversarial Network (GAN)-based methods lack generalization, producing inauthentic and non-diverse images.
• Diffusion models show superior performance in visual tasks, surpassing GANs.

Purpose of the Study:

• To pioneer diffusion models for weather image translation using a novel Instruction-driven Multi-Weather Translation (InstructWT) framework.
• To enhance the authenticity and diversity of translated weather images.
• To improve the performance of semantic segmentation algorithms under various weather conditions.

Main Methods:

• Developed InstructWT based on the InstructPix2Pix large image editing model, leveraging its zero-shot generalization.
• Implemented prompt engineering for a user-friendly instruction set and introduced a weather intensity factor for precise control.
• Utilized weather correlation-based blended editing and physically based rendering for rain and snow to preserve scene layout and enhance realism.

Main Results:

• InstructWT significantly outperforms existing methods on the Cityscapes dataset in authenticity and fidelity.
• Achieved high Contrastive Language-Image Pre-Training (CLIP) image embedding cosine similarity (0.8302) and directional CLIP similarity (0.1598).
• Semantic segmentation algorithms fine-tuned with InstructWT-augmented data demonstrated substantial performance improvements in complex weather scenarios.

Conclusions:

• InstructWT represents a significant advancement in diffusion-based weather image translation, offering improved realism and diversity.
• The method effectively addresses the limitations of GAN-based approaches and enhances the utility of synthetic data for downstream tasks.
• InstructWT shows strong potential for applications requiring realistic multi-weather image generation and robust computer vision models.