Deblur or denoise: the role of an aperture in lens and neural network co-design

Co-design methods have been introduced to jointly optimize various optical systems along with neural network processing. In the literature, the aperture is generally a fixed parameter although it controls an important trade-off between the depth of focus, the dynamic range, and the noise level in an image. In contrast, we include aperture in co-design by using a differentiable image formation pipeline that models the effect of the aperture on the image noise, dynamic, and blur. We validate this pipeline on examples of image restoration and extension of the depth of focus. These simple examples illustrate the importance of optimizing the aperture in the co-design framework.