FreeEdit: Mask-Free Reference-Based Image Editing With Multi-Modal Instruction

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• User-specified visual concepts offer more precise intent than text for image editing.
• Existing methods struggle with reference-based editing due to dataset limitations and manual mask requirements.

Purpose of the Study:

• To introduce FreeEdit, a novel approach for reference-based image editing guided by multi-modal instructions.
• To enhance the reconstruction of fine-grained details from reference images.
• To develop a suitable dataset for reference-based image editing tasks.

Main Methods:

• Leveraging a multi-modal instruction encoder to interpret language instructions for guiding the editing process.
• Introducing the Decoupled Residual Refer-Attention (DRRA) module for integrating reference details without disrupting self-attention.
• Curating the FreeEdit dataset using a novel twice-repainting scheme for image triplets (before/after editing, instructions, reference image).

Main Results:

• FreeEdit achieves high-quality zero-shot editing through phased training and quality tuning.
• The DRRA module effectively integrates fine-grained reference features.
• Extensive experiments demonstrate FreeEdit's superiority over existing methods across various editing tasks (addition, replacement, deletion).

Conclusions:

• FreeEdit provides an effective and user-friendly solution for reference-based image editing.
• The approach implicitly locates editing areas, removing the need for manual masks.
• The curated FreeEdit dataset facilitates further research in this domain.