Analog in-memory computing attention mechanism for fast and energy-efficient large language models.

Area of Science:

• Computer Science
• Electrical Engineering
• Artificial Intelligence

Background:

• Transformer networks and self-attention mechanisms are fundamental to large language models (LLMs).
• Current generative transformers face latency and energy bottlenecks due to data movement between graphics processing units (GPUs) and static random-access memory (SRAM).

Purpose of the Study:

• To introduce a custom self-attention in-memory computing architecture to overcome existing latency and energy limitations in generative transformers.
• To enable efficient, low-power computation for LLMs.

Main Methods:

• Developed a novel in-memory computing architecture utilizing charge-based memory gain cells for parallel analog dot-product computation.
• Designed an initialization algorithm to address non-idealities in analog gain-cell circuits, enabling performance comparable to pre-trained models like GPT-2 without full retraining.

Main Results:

• The proposed architecture reduces attention latency by up to two orders of magnitude and energy consumption by up to four orders of magnitude compared to GPU-based systems.
• Achieved text-processing performance comparable to GPT-2 using the novel architecture and initialization algorithm.

Conclusions:

• The custom in-memory computing architecture offers a substantial advancement for ultrafast and low-power generative transformers.
• This approach paves the way for more efficient and accessible large language model deployment.