大量切换的基于memristor的内存计算模块用于深度神经网络训练
Yuting Wu1, Qiwen Wang1, Ziyu Wang1
1Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI, 48109, USA.
Advanced materials (Deerfield Beach, Fla.)
|September 25, 2023
概括
本研究介绍了一种用于使用基于memristor的内存计算 (CIM) 模块的深度神经网络 (DNN) 的混合精度训练方案. 这种方法加速了培训,并实现了与传统方法相比的高精度.
科学领域:
- 电气工程 电气工程
- 计算机科学 计算机科学
- 材料科学 材料科学 材料科学
背景情况:
- 深度神经网络 (DNN) 需要大量的计算资源进行训练.
- 基于memristor的内存计算 (CIM) 为有效的DNN推断提供了潜力,但由于设备非线性和精度限制,在训练中面临挑战.
研究的目的:
- 实验实施和验证使用基于memristor的CIM模块对DNN进行混合精度训练方案.
- 解决基于CIM的DNN培训中的挑战,包括非线性重量更新,设备变化和低精度.
主要方法:
- 开发了一个混合精度训练方案,使用低精度CIM模块进行加速向量矩阵乘法 (VMM) 和高精度数字单元来积累重量更新.
- 只有当累积的重量变化超过一个值时,Memristor设备才会更新.
- 该方案在集成模拟CIM模块和数字子系统的芯片系统上实施.
主要成果:
- 拟议的方案显示了LeNet培训的快速融合,达到97.73%的准确性.
- 使用现实的硬件参数进行的评估证实了CIM模块对于高效的混合精度DNN训练的有效性.
- 在芯片上训练的模型显示出对硬件变化的稳定性,使其能够直接部署在CIM推理芯片上.
结论:
- 使用基于memristor的CIM模块进行混合精度训练,可实现高效准确的DNN训练.
- 这种方法减轻了硬件变化,并简化了在CIM推理硬件上部署训练模型.
- 开发的芯片上的系统促进了先进的人工智能模型的实际实施.
相关概念视频
01:19Understanding Memory
358
Memory is the retention of information or experiences over time, facilitated through three main processes: encoding, storage, and retrieval. Encoding is the process of inputting information into the memory system. For instance, when listening to a lecture, watching a play, reading a book, or having a conversation, the brain is actively encoding information. This initial stage involves transforming sensory input into a form that can be processed and stored by the brain. Various factors, such as...
358
01:23System of Memory
6.1K
Memory is categorized into three major systems: sensory memory, short-term memory (STM), and long-term memory (LTM). These systems differ in their capacity and the duration for which they can hold information. Sensory memory captures raw sensory input from the environment, holding it for just a few seconds or less. For example, on hearing a brief, loud sound, like a car horn honking, the sound seems to linger in the mind for a moment even after it stops. This is an instance of sensory memory...
6.1K
01:25MOS Capacitor
825
A Metal-Oxide-Semiconductor (MOS) capacitor is a fundamental structure used extensively in semiconductor device technology, particularly in the fabrication of integrated circuits and MOSFETs (metal-oxide-semiconductor field-effect transistors). The MOS capacitor consists of three layers: a metal gate, a dielectric oxide, and a semiconductor substrate.
The metal gate is typically made from highly conductive materials such as aluminum or polysilicon. Beneath the metal gate lies a thin layer of...
The metal gate is typically made from highly conductive materials such as aluminum or polysilicon. Beneath the metal gate lies a thin layer of...
825
01:25Neural Circuits
1.3K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.3K
01:26Higher Mental Functions of Brain: Learning and Memory
861
Memory is one of the most vital higher mental functions of the brain. Memory is closely related to learning because it enables us to retain information and experiences from our past to use them in our present life. It also helps us to remember facts, events, and skills, such as riding a bike or swimming. There are two types of memory — declarative memory, which involves memorizing facts or events, and procedural memory, which enables us to remember how to do something like writing or...
861
01:22MOSFET: Enhancement Mode
374
Enhancement-mode MOSFETs are pivotal components in electronics, distinguished by their capacity to act as highly efficient switches. They are part of the larger family of metal-oxide Semiconductor Field-Effect Transistors (MOSFETs). They are available in two types: p-channel and n-channel, each tailored to specific polarity operations.
In their basic form, enhancement-mode MOSFETs are typically non-conductive when the gate-source voltage (Vgs) is zero. This default 'off' state means no...
In their basic form, enhancement-mode MOSFETs are typically non-conductive when the gate-source voltage (Vgs) is zero. This default 'off' state means no...
374