On the computational power of threshold circuits with sparse activity.

Area of Science:

• Computational neuroscience
• Theoretical computer science
• Artificial intelligence

Background:

• Threshold circuits (McCulloch-Pitts neurons/perceptrons) are simplified neural models but require high activity, unlike sparse biological neural networks.
• Previous complexity measures (size, depth) for threshold circuits do not reflect biological neural activity patterns.

Purpose of the Study:

• To introduce and investigate energy complexity as a novel measure for threshold circuits.
• To demonstrate that minimizing energy complexity leads to sparse neural activity.
• To explore the computational power and structural properties of energy-optimized threshold circuits.

Main Methods:

• Theoretical analysis of threshold circuits.
• Introduction of 'energy complexity' as a new metric.
• Proof of restructuring capabilities for polynomial-size threshold circuits with O(log n) entropy.
• Development of 'entropy of circuit states' as a novel complexity measure.

Main Results:

• Minimizing energy complexity yields threshold circuits with sparse activity, closer to biological neural networks.
• All polynomial-size threshold circuits with O(log n) entropy can be restructured to reduce energy complexity.
• A novel circuit complexity measure, 'entropy of circuit states,' is introduced.
• Polynomial-size threshold circuits with O(log n) entropy can be simulated by depth-3 circuits.

Conclusions:

• Minimizing energy complexity results in circuits with distinct structures and sparse activity, potentially mimicking biological neural systems.
• Sparse activity in threshold circuits does not limit but rather enhances computational power.
• The findings suggest a new direction for designing artificial neural networks that are more biologically plausible and computationally efficient.