在预训练的早期退出神经网络中,零时间浪费
Bartosz Wójcik1, Marcin Przewiȩźlikowski1, Filip Szatkowski2
1Faculty of Mathematics and Computer Science, Jagiellonian University, Poland; Doctoral School of Exact and Natural Sciences, Jagiellonian University, Poland; IDEAS NCBR, Poland.
概括
零时间浪费 (ZTW) 通过重复使用中间预测来增强深度学习模型,减少浪费计算. 这种新的方法显著改善了准确性-推理时间权衡,以实现高效的模型处理.
科学领域:
- 计算机科学 计算机科学
- 人工智能的人工智能
- 机器学习 机器学习
背景情况:
- 缩短大型深度学习模型的处理时间对于现实应用至关重要.
- 早期退出方法使用内部分类器 (ICs) 通过早期预测简单的例子来减少推理时间.
- 目前的方法浪费了IC的计算,这些IC不会提前退出.
研究的目的:
- 介绍零时间浪费 (ZTW),一种新的方法来消除早期退出方法中的浪费计算.
- 在深度学习模型中改善准确度-推断时间权衡.
主要方法:
- ZTW重新使用了之前内部分类器 (IC) 的预测.
- 在IC之间增加了直接连接.
- 之前的产品以合奏式的方式结合在一起.
主要成果:
- 与现有的早期退出方法相比,ZTW在推断时间权衡方面表现出更高的准确性.
- 在ImageNet上取得了显著的改进,在16种场景中的11种场景中表现优于基线.
- 在较低的计算预算中显示了高达5个百分点的改进.
结论:
- ZTW有效地重用中间预测,最大限度地减少计算浪费.
- 该方法为深度学习模型提供了准确性和推断速度之间的更好的平衡.
- ZTW为大型深度学习模型的高效处理提供了一个有前途的解决方案.
相关概念视频
Parameters Affecting Nonlinear Elimination: Zero-Order Input, First-Order Absorption and Two-Compartment Model
70
Drugs administered through various routes can lead to nonlinear elimination, resulting in complex pharmacokinetic behaviors crucial to understanding efficacious drug dosing.
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
When a drug is administered through a constant intravenous infusion and eliminated via nonlinear pharmacokinetics, it follows zero-order input. For example, oral drugs undergo first-order absorption upon administration and are eliminated through nonlinear pharmacokinetics.
In the case of subcutaneously administered drugs,...
70
Avoidance Learning and Learned Helplessness
1.7K
Avoidance learning and learned helplessness are critical concepts in understanding behavioral responses to negative stimuli.
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
Avoidance learning occurs when an organism learns that a specific behavior can prevent an unpleasant outcome. For example, a student who receives a bad grade may start studying harder to avoid future poor grades. This behavior persists even when the negative outcome is no longer present. Avoidance learning is powerful because it maintains behavior in the absence of the...
1.7K
Neural Circuits
1.3K
Neural circuits and neuronal pools are two of the main structures found in the nervous system. Neural circuits are networks of neurons that work together to carry out a specific task or process. They consist of interconnected neurons and glial cells, which provide structural and metabolic support.
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
Neuronal pools are collections of nerve cells with similar functions and interact through chemical and electrical signals. These pools include both interneurons (the central neural circuit nodes that...
1.3K
Survival Tree
89
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
89
Reducing Line Loss
156
In a three-phase circuit, line loss is an indicator of energy dissipated as heat due to the resistance of transmission lines. To address this, incorporating transformers into the system—a step-up transformer at the source and a step-down transformer at the load—is a strategic solution. Two three-phase transformers are introduced to improve this.
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
With a step-up transformer at the source, the voltage is increased, thereby reducing the current in the transmission lines since power loss...
156
End Point Prediction: Gran Plot
351
A Gran plot is used to predict the equivalence volume or endpoint of a potentiometric or acid-base titration without reaching the endpoint. Typically, titration data is collected as a function of the titrant's volume up to a point less than the equivalence volume and then transformed into a linear format. The straight line is extended to the x-axis, indicating the necessary titrant volume to achieve the equivalence point.
For potentiometric titration, the Gran plot is created by plotting...
For potentiometric titration, the Gran plot is created by plotting...
351


