ProReGen: Progressive Residual Generation under Attribute Correlations

Attribute correlations in the training data will compromise the ability of a deep generative model (DGM) to synthesize images with under-represented attribute combinations (i.e., minority samples). Existing approaches mitigate this by data re-sampling to remove attribute correlations seen by the DGM, using a classifier to provide pseudo-supervision on generated counterfactual samples, or incorporating inductive bias to explicitly decompose the generation into independent submechanisms. We present ProReGen, a progressive residual generation approach inspired by the classical Robinson's transformation, to partial out from an image attribute $x_2$ its component $m\left(x_1\right)$ that is predictable by other image attributes $x_1$ , and the residual $\gamma =x_2-m\left(x_1\right)$ that is not. This simplifies the problem of learning a DGM $g\left(x_1,x_2\right)$ conditioned on correlated inputs, to learning $\tilde{g}\left(x_1,\gamma \right)$ conditioned on orthogonal inputs. It further allows us to progressively learn $\tilde{g}$ by first shifting the burden to abundant majority samples to learn $\tilde{g}\left(x_1,\gamma =0\right)$ , and then expanding it with additional layers $g_{\text{res}}$ to resolve its difference to $\tilde{g}\left(x_1,\gamma \right)$ using residual attribute $\gamma$ on limited minority samples. On three benchmark datasets with varying strengths of attribute correlation and one dataset with natural attribute correlation, we demonstrate that ProReGen-with input orthogonalization and progressive residual learning-improved the correctness of minority generations compared to existing strategies.