Estimation of energy-dissipation lower bounds for neuromorphic learning in memory

Area of Science:

• * Neuromorphic Engineering
• * Computational Optimization
• * Artificial Intelligence

Background:

• * Neuromorphic optimizers utilize parallel parameter updates for complex problems.
• * Existing designs face energy challenges: memory wall, update wall, and consolidation wall.
• * Compute-in-memory and learning-in-memory paradigms offer solutions to these energy bottlenecks.

Purpose of the Study:

• * To derive theoretical energy-to-solution estimates for an ideal neuromorphic optimizer.
• * To analyze the energy efficiency of optimizers by modulating physical memory energy barriers.
• * To demonstrate practical applicability for large-scale AI workloads.

Main Methods:

• * Theoretical derivation of energy-to-solution metrics.
• * Analysis of out-of-equilibrium thermodynamics of learning.
• * Model-agnostic estimation based on update operations, model size, convergence speed, and solution precision.

Main Results:

• * Theoretical estimates for the energy-to-solution metric of an ideal neuromorphic optimizer.
• * Energy efficiency is determined by memory update/consolidation dynamics matching optimization dynamics.
• * Lower bounds for energy-to-solution metrics are estimated for large-scale AI workloads.

Conclusions:

• * An ideal neuromorphic optimizer can significantly reduce energy consumption.
• * The derived estimates are model-agnostic, applicable across various optimization tasks.
• * This work provides a framework for evaluating energy efficiency in neuromorphic computing for AI.