Locally linear attributes of ReLU neural networks

Area of Science:

• Computational Neuroscience
• Machine Learning Theory
• Artificial Intelligence

Background:

• Rectified Linear Unit (ReLU) neural networks are fundamental in deep learning, acting as continuous piecewise linear functions.
• The network's weights define a partitioning of the input space into convex polytopes, each with a unique affine mapping.

Purpose of the Study:

• To investigate the geometric structure of input space partitioning by ReLU networks.
• To understand how network complexity can be reduced by analyzing and potentially merging similar regions.

Main Methods:

• Analyzing the partitioning of input space induced by ReLU network weights.
• Developing intuition through simple problems to understand region behavior and reduction.
• Applying findings to validate intuitions on networks trained on MNIST dataset.

Main Results:

• The partitioning of input space by ReLU networks creates distinct regions, each governed by a specific affine map.
• Similar structural patterns were observed across different ReLU networks, even those trained on complex tasks like MNIST.
• The analysis suggests a potential for reducing the complexity of trained neural networks.

Conclusions:

• The geometric structure of ReLU networks offers insights into their behavior and potential for optimization.
• Understanding the partitioning and affine mappings is key to analyzing and simplifying neural network architectures.
• This research opens avenues for developing more efficient and interpretable deep learning models.