Trade-Offs Between Energy and Depth of Neural Networks.

Area of Science:

• Computational complexity theory
• Artificial intelligence
• Machine learning

Background:

• Discretized neural networks, including threshold circuits, are crucial for efficient computation.
• Understanding the interplay of computational resources like size, depth, weight, and energy is vital for optimizing these networks.
• Energy, defined as maximum nonzero gate activations, offers a novel complexity measure inspired by sparse coding.

Purpose of the Study:

• To analyze the relationship between computational resources (size, depth, weight, energy) and the complexity of Boolean functions computed by discretized neural networks.
• To establish theoretical bounds on the computational power of these networks based on their resource constraints.
• To identify trade-offs between different computational resources, particularly depth and energy.

Main Methods:

• Derivation of a theoretical bound relating the rank of a Boolean function to the energy, depth, size, and weight of a threshold circuit computing it.
• Analysis of a specific Boolean function (CDn) to illustrate the derived bound and its implications.
• Extension of the analysis to other discretized neural network models, such as ReLU and sigmoid circuits.

Main Results:

• A key finding establishes that log(rk(f)) ≤ ed(log s + log w + log n) for a threshold circuit C computing a Boolean function f.
• For the CDn function, the inequality n/2 ≤ ed(log s + log w + log n) demonstrates a linear relationship between function complexity and the product of energy and depth.
• Similar trade-offs between depth and energy were proven for discretized ReLU and sigmoid circuits.

Conclusions:

• A significant trade-off exists between the depth and energy of discretized neural networks for computing Boolean functions.
• Increasing network depth can enhance the ability to learn sparse representations, especially when hardware resources like neuron count and weight resolution are limited.
• These findings provide theoretical insights into the efficient design and resource allocation for discretized neural networks.