Perovskite Neuromorphic Engine for Transformer Architectures

Area of Science:

• Materials Science
• Computer Engineering
• Neuroscience

Background:

• Memristive computing offers efficient analog multiply-and-accumulation (MAC) operations for artificial neural networks (ANNs).
• Current memristive approaches face inefficiencies in advanced network structures due to analog-digital data exchange.
• Transformer ANNs, crucial for advanced AI, require complex operations that challenge existing memristive implementations.

Purpose of the Study:

• To develop a perovskite memristive computing unit capable of performing all mathematical operations for Transformer ANNs in the analog domain.
• To demonstrate the feasibility of a fully analog neuromorphic engine for advanced AI tasks.
• To overcome the limitations of data conversion in current memristive computing architectures.

Main Methods:

• Fabrication of a perovskite memristive computing unit using vapor deposition, enabling reconfigurability and nonlinearity.
• Implementation of a prototypical attention module using memristive cells configured for dynamic MAC, activation, and softmax functions.
• Construction and testing of a multi-layer Transformer network utilizing cascaded attention modules for real-world tasks.

Main Results:

• The developed memristive unit successfully performed all necessary operations for Transformer ANNs in the analog domain.
• A neuromorphic engine based on this unit achieved performance comparable to GPU acceleration on RGB-T tracking and visual question answering tasks.
• The memristive engine demonstrated a 98.3% reduction in energy consumption (1.7% of GPU) and a 58-fold increase in power efficiency.

Conclusions:

• The perovskite memristive computing unit offers a pathway to highly efficient and accurate hardware acceleration for advanced ANNs, particularly Transformer models.
• Fully analog processing in memristive devices eliminates data conversion bottlenecks, paving the way for next-generation neuromorphic computing.
• This work highlights the potential of memristive devices in realizing energy-efficient AI hardware for complex computational tasks.