Optimal linear compression under unreliable representation and robust PCA neural models

In a typical linear data compression system the representation variables resulting from the coding operation are assumed totally reliable and therefore the solution in the mean-squared-error sense is an orthogonal projector to the so-called principal component subspace. When the representation variables are contaminated by additive noise which is uncorrelated with the signal, the problem is called noisy principal component analysis (NPCA) and the optimal MSE solution is not a trivial extension of PCA. We first show that the problem is not well defined unless we impose explicit or implicit constraints on either the coding or the decoding operator. Second, orthogonality is not a property of the optimal solution under most constraints. Third, the signal components may or may not be reconstructed depending on the noise level. As the noise power increases, we observe rank reduction in the optimal solution under most reasonable constraints. In these cases it appears that it is preferable to omit the smaller signal components rather than attempting to reconstruct them. This phenomenon has similarities with classical information theoretical results, notably the water-filling analogy, found in parallel additive Gaussian noise channels. Finally, we show that standard Hebbian-type PCA learning algorithms are not optimally robust to noise, and propose a new Hebbian-type learning algorithm which is optimally robust in the NPCA sense.