End-to-end sensor and neural network design using differential ray tracing

In this paper we propose a new method to jointly design a sensor and its neural-network based processing. Using a differential ray tracing (DRT) model, we simulate the sensor point-spread function (PSF) and its partial derivative with respect to any of the sensor lens parameters. The proposed ray tracing model makes no thin lens nor paraxial approximation, and is valid for any field of view and point source position. Using the gradient backpropagation framework for neural network optimization, any of the lens parameter can then be jointly optimized along with the neural network parameters. We validate our method for image restoration applications using three proves of concept of focus setting optimization of a given sensor. We provide here interpretations of the joint optical and processing optimization results obtained with the proposed method in these simple cases. Our method paves the way to end-to-end design of a neural network and lens using the complete set of optical parameters within the full sensor field of view.