Energy complexity of recurrent neural networks

Area of Science:

• Computational Neuroscience
• Theoretical Computer Science
• Artificial Intelligence

Background:

• A novel energy complexity measure was recently developed for feedforward perceptron networks.
• This measure is motivated by biological neuron energy consumption and sparse neural activity.
• Biological neurons expend more energy transmitting spikes than remaining inactive.

Purpose of the Study:

• To investigate the energy complexity of recurrent neural networks.
• To analyze the energy cost of simulating deterministic finite automata using recurrent networks.
• To establish a time-energy trade-off in recurrent network computations.

Main Methods:

• Analyzing recurrent neural network simulations of deterministic finite automata.
• Quantifying the number of active neurons as a measure of energy complexity.
• Deriving theoretical bounds on simulation time overhead and energy consumption.

Main Results:

• Any deterministic finite automaton with m states can be simulated by an optimal-sized recurrent neural network.
• The simulation achieves a time overhead of O(log s) per input bit with energy O(e), where e satisfies specific bounds.
• A lower bound on simulation energy is established for certain time overheads.

Conclusions:

• Recurrent neural networks exhibit a demonstrable time-energy trade-off.
• The findings provide insights into the energy efficiency of neural network computations for sequential tasks.
• This work extends energy complexity analysis to recurrent network architectures.