SegPainter: User-Controllable Face Inpainting via Mask-Aware Semantic Segmentation-Guided Mamba

With the growing demand for face editing applications, face inpainting has become an increasingly important subfield within image inpainting research. While many existing methods use semantic segmentation guidance, they apply uniform weighting across all regions of the map. Since the missing areas of the image lack meaningful features, this uniform treatment provides insufficient guidance in missing areas, leading to unrealistic or structurally incoherent results. Moreover, these methods generally lack the capacity to adapt inpainting results to individual user preferences, thereby limiting their effectiveness in personalized face editing. To address these limitations, we propose SegPainter, a Mamba-based architecture for user-controllable face inpainting that enables customized restoration guided by user-defined semantic segmentation maps generated using the Image Segmentation Annotation Tool (ISAT) integrated with Meta's Segment Anything Model (SAM). Specifically, we propose Hard Mask Soft One-Hot Encoding (HMSOE) to adaptively weight regions in the segmentation map based on whether they correspond to known or missing areas of the masked image. This strategy amplifies semantic guidance in missing regions while attenuating it in known regions to avoid over-constraining existing content. We further introduce Semantic-Guided State Space Model (SG-SSM) to dynamically modulate the Mamba layer with semantic features, adapting guidance to the masked image. To enhance the quality of inpainting results, we also propose Tri-Scan Inspection (TSI), a scanning mechanism designed to capture both global and local dependencies while preserving spatial continuity and facial structure. Extensive experiments on the CelebAMask-HQ and FFHQ datasets demonstrate that our framework outperforms state-of-the-art methods, producing sharper and more semantically consistent face inpainting results.