Towards continual low-light image enhancement through causal inference

Low-light image enhancement focuses on improving the visual perceptual quality and restoring structure details for images captured from poor lightness conditions. The existing approaches often perform well under the identical lightness distribution with the training dataset. We address a continual low-light image enhancement task for the first time to challenge their generalization ability. Specifically, a lightness enhancement network learns a sequence of tasks with different lightness distributions. While the model's generalization performance has been improved, we still identify the presence of catastrophic forgetting in this scenario, which prompts us to further disclose the intrinsic reason from theory. Therefore, we leverage causal inference theory to construct a structural causal model, identifying confounding variables and mitigating their adverse effects through backdoor adjustment, which facilitates the model to learn invariant representation for continual low-light image enhancement. In particular, the Rehearsal-based Invariant Structure Regularizer is employed to enlarge the available value set of the variable for backdoor adjustment. Moreover, a novel Channel Fourier Transform-based Self-Attention module is proposed to enhance the model's generalization ability toward low-frequency information and facilitate the precise estimation of the causal effect. Extensive experiments show the effectiveness in alleviating catastrophic forgetting and superior generalization performance of our method on continuous low-light image enhancement benchmarks.