改进预训练的语言模型 微调与噪声稳定性 调整调整
IEEE transactions on neural networks and learning systems
|November 30, 2023
概括
层层的噪声稳定规范化 (LNSR) 通过在微调过程中添加噪声来增强预训练的语言模型,提高了像问答这样的复杂任务的概括性. 这种方法有效地打击了自然语言处理中的过度插入.
科学领域:
- 自然语言处理 (NLP) 是一种自然语言处理.
- 机器学习 机器学习
- 深度学习 (Deep Learning) 是一种深度学习.
背景情况:
- 预训练的语言模型 (PLM) 已经有了先进的NLP.
- 微调的PLM可能会导致过度拟合和糟糕的概括性,这是由于模型的复杂性和有限的数据.
研究的目的:
- 引入一种新的微调框架,分层噪声稳定性调节 (LNSR),以减轻PLM中的过度合.
- 提高语言模型的概括性和域概括能力.
主要方法:
- 在表示空间中,LNSR会以高斯式或多元噪声扰乱神经网络输入.
- 该方法规范了语言模型中的每个层的输出.
- 理论和实验分析验证了提出的方法.
主要成果:
- 它的性能优于包括L2-SP,Mixout,FreeLB和SMART在内的最先进的方法.
- 该框架在文本分类和更具挑战性的问答任务方面表现出有效性.
- 经验结果显示,语言模型的域概括能力得到了改善.
结论:
- LNSR是一种有效的微调策略,可以提高PLM的通用性.
- 该方法提供了一个强大的解决方案,用于过度适应NLP任务.
- 在各种下游应用中,LNSR显示出提高模型性能的前景.
更多相关视频
相关概念视频
Improving Translational Accuracy
2.6K
2.6K
Regression Toward the Mean
6.3K
Regression toward the mean (“RTM”) is a phenomenon in which extremely high or low values—for example, and individual’s blood pressure at a particular moment—appear closer to a group’s average upon remeasuring. Although this statistical peculiarity is the result of random error and chance, it has been problematic across various medical, scientific, financial and psychological applications. In particular, RTM, if not taken into account, can interfere when...
6.3K
Residuals and Least-Squares Property
7.4K
The vertical distance between the actual value of y and the estimated value of y. In other words, it measures the vertical distance between the actual data point and the predicted point on the line
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
If the observed data point lies above the line, the residual is positive, and the line underestimates the actual data value for y. If the observed data point lies below the line, the residual is negative, and the line overestimates the actual data value for y.
The process of fitting the best-fit...
7.4K
Calibration Curves: Linear Least Squares
1.3K
A calibration curve is a plot of the instrument's response against a series of known concentrations of a substance. This curve is used to set the instrument response levels, using the substance and its concentrations as standards. Alternatively, or additionally, an equation is fitted to the calibration curve plot and subsequently used to calculate the unknown concentrations of other samples reliably.
For data that follow a straight line, the standard method for fitting is the linear...
For data that follow a straight line, the standard method for fitting is the linear...
1.3K
Random and Systematic Errors
11.0K
Scientists always try their best to record measurements with the utmost accuracy and precision. However, sometimes errors do occur. These errors can be random or systematic. Random errors are observed due to the inconsistency or fluctuation in the measurement process, or variations in the quantity itself that is being measured. Such errors fluctuate from being greater than or less than the true value in repeated measurements. Consider a scientist measuring the length of an earthworm using a...
11.0K
RNA Stability
33.6K
Intact DNA strands can be found in fossils, while scientists sometimes struggle to keep RNA intact under laboratory conditions. The structural variations between RNA and DNA underlie the differences in their stability and longevity. Because DNA is double-stranded, it is inherently more stable. The single-stranded structure of RNA is less stable but also more flexible and can form weak internal bonds. Additionally, most RNAs in the cell are relatively short, while DNA can be up to 250 million...
33.6K


