A Super Memory Processing Unit Based on 3D Stacking and Hybrid Bonding for High-Efficiency AI Computing

Area of Science:

• Computer Engineering
• Memory Systems
• Artificial Intelligence Hardware

Background:

• DRAM-based in-memory computing (DRAM-PIM) offers efficiency but faces limitations in general processing power.
• Existing DRAM-PIM designs struggle with restricted computational capabilities due to shared package areas or over-customized circuits.

Purpose of the Study:

• To propose a novel Super Memory Processing Unit (SMPU) to overcome the computational limitations of DRAM-PIM.
• To enhance data processing capabilities and system performance for diverse AI applications.

Main Methods:

• Utilized Hybrid Bonding technology to 3D-stack DRAM with many-core computational clusters.
• Implemented a dual-channel, fine-grained computational cluster at the logical computing layer.
• Integrated a memory space allocation and parsing controller with standard DDR protocols for host compatibility.

Main Results:

• Achieved large on-chip bandwidth (2 TB/s for an 8-bank system) via copper interconnects, breaking the memory wall bottleneck.
• Demonstrated significant performance improvements: up to 5.1x for ResNet50 and 27.43x for Llama 7B.
• Reduced system power consumption by 71.6% (ResNet50) to 77.8% (Llama 7B) compared to baseline systems.

Conclusions:

• The SMPU effectively enhances DRAM-PIM by providing general and powerful data processing capabilities.
• The proposed architecture offers flexible, high-bandwidth, and power-efficient computation for AI models.
• SMPU facilitates seamless integration into existing systems, paving the way for advanced in-memory computing solutions.