对大型和多样化的深度学习推断工作负载进行硬件意识培训,使用基于内存计算的加速器
Malte J Rasch1, Charles Mackin2, Manuel Le Gallo3
1IBM Research, TJ Watson Research Center, Yorktown Heights, NY, USA. malte.rasch@ibm.com.
Nature communications
|August 30, 2023
概括
模拟内存计算可以实现深度学习任务的高精度. 硬件意识的再培训使神经网络能够保持性能,尽管硬件不完美,特别是对于经常性网络.
科学领域:
- 机器学习是对硬件有意识的.
- 节能计算架构的节能计算架构.
- 深度学习加速加速
背景情况:
- 模拟内存计算 (AIMC) 通过执行矩阵向量乘法提供了节能的深度学习加速.
- 在AIMC中非理想的设备特征可能导致非决定性或非线性运算,可能会降低推理准确性.
- 现有的再培训方法可能无法完全解释模拟硬件中存在的复杂的非理想性.
研究的目的:
- 开发一种硬件意识的再培训方法,用于系统地评估模拟内存计算精度.
- 研究深度神经网络对AIMC中的各种非理想性的敏感性和稳定性.
- 为了证明深度神经网络可以达到与使用AIMC的浮点实现相似的准确性.
主要方法:
- 开发了一种硬件意识的再培训方法,以优化AIMC的深度神经网络.
- 整合了一个现实的横杆模型来模拟AIMC的非理想性.
- 分析了多种网络拓,包括卷积神经网络 (convnets),循环神经网络 (RNNs) 和变压器.
- 对一系列非理想条件进行了敏感性分析.
主要成果:
- 许多大型深度神经网络,包括convnets,RNNs和变压器,可以成功地重新训练,以实现与浮点实现相比的iso-accuracy.
- 影响输入或输出的非理想性对准确度的影响比影响权重的非理想性更大.
- 经常性神经网络表现出对所有被调查的非理想性具有特别的稳定性.
结论:
- 硬件意识的再培训有效地减轻了深度学习的模拟内存计算中的精度退化.
- 输入/输出噪声显著影响AIMC的准确性,突出了硬件和算法共同设计的领域.
- 循环网络是强大而准确的模拟内存计算应用程序的有希望的架构.
相关概念视频
01:20Parallel Processing
179
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
179
01:19Understanding Memory
535
Memory is the retention of information or experiences over time, facilitated through three main processes: encoding, storage, and retrieval. Encoding is the process of inputting information into the memory system. For instance, when listening to a lecture, watching a play, reading a book, or having a conversation, the brain is actively encoding information. This initial stage involves transforming sensory input into a form that can be processed and stored by the brain. Various factors, such as...
535
01:17Ampere-Maxwell's Law: Problem-Solving
667
A parallel-plate capacitor with capacitance C, whose plates have area A and separation distance d, is connected to a resistor R and a battery of voltage V. The current starts to flow at t = 0. What is the displacement current between the capacitor plates at time t? From the properties of the capacitor, what is the corresponding real current?
To solve the problem, we can use the equations from the analysis of an RC circuit and Maxwell's version of Ampère's law.
For the first part of...
To solve the problem, we can use the equations from the analysis of an RC circuit and Maxwell's version of Ampère's law.
For the first part of...
667
01:23System of Memory
6.2K
Memory is categorized into three major systems: sensory memory, short-term memory (STM), and long-term memory (LTM). These systems differ in their capacity and the duration for which they can hold information. Sensory memory captures raw sensory input from the environment, holding it for just a few seconds or less. For example, on hearing a brief, loud sound, like a car horn honking, the sound seems to linger in the mind for a moment even after it stops. This is an instance of sensory memory...
6.2K
02:07Improving Translational Accuracy
11.6K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
11.6K
01:19Machines
302
Machines are complex structures consisting of movable, pin-connected multi-force members that work together to transmit forces. One example of a machine is the cutting plier, which is used to cut wires by applying forces to its handles. When equal and opposite forces are exerted on the handles of the cutting plier, they cause the cutting edges to come together and apply equal and opposite reaction forces on the wire, which are greater than the applied forces.
A free-body diagram of the...
A free-body diagram of the...
302