Characterization of local linearity in deep learning CT denoising networks

Deep learning algorithms for image processing have been proposed for a wide range of applications. In particular, deep learning CT denoising has demonstrated superior ability to reduce noise compared to conventional methods. However, the nonlinear nature of these algorithms makes it difficult to inspect and visualize how the model accomplishes the denoising task.

We propose a local linearization approach to characterize and analyze the behavior of deep learning algorithms. Example analysis was performed for a classic denoising architecture with convolutional neural network and ReLU activation functions, and was extended to GELU activation and a transformer-based architecture.

We defined strictly local linear regions as the input space over which the activation patterns of all ReLU activation functions are the same. The extent of these regions was investigated by perturbing clinically relevant features of a lung lesion. We further defined approximately local linear regions as the input space where Jacobians are similar and linear approximation results in small error. We developed an algorithm to partition input space over a range of lesion and background features into approximately local linear regions, and separately quantified the contribution of the noise realization and anatomical background to this approximation error. The behavior of networks trained with different levels of denoising, as well as alternative architectures, was compared.

The extent of strictly local linear regions was found to be very small. Perturbations on the scale of ${10}^{-8}{\text{mm}}^{-1}$ were enough to push the network to a neighboring strictly local linear region. However, the Jacobians over a large number of strictly local linear regions were found to be highly similar, which suggests that the network behaves approximately as a piecewise linear system. The partitioning algorithm identified a small number of approximately local linear regions within input spaces characterized by clinically relevant features; this number was similar for the GELU variant but increased for networks trained with more aggressive noise reduction and for a transformer-based architecture, while Jacobians transferred across anatomical backgrounds also increased approximation error.

The local linearity analysis framework can be a useful tool for elucidating the behavior of neural networks that are traditionally considered as "black-boxes." Preliminary results suggest that the framework can be applied to other differentiable architectures, subject to empirical validation, to characterize these highly nonlinear, data-dependent algorithms over input spaces of interest.