PME:用于推系统的基于修剪的多尺寸嵌入
Zirui Liu1, Qingquan Song2, Li Li3
1Computer Science Department, Rice University, Houston, TX, United States.
Frontiers in big data
|July 3, 2023
概括
本研究介绍了一种基于修剪的多尺寸嵌入 (PME) 框架,以优化推模型. 通过削减影响较小的尺寸,PME有效地减少了嵌入参数和内存使用,而不会牺牲性能.
科学领域:
- 机器学习 机器学习
- 人工智能的人工智能
- 数据科学数据科学数据科学
背景情况:
- 传统的嵌入技术在推模型中使用所有特征的固定大小,导致潜在的内存低效率.
- 现有的定制嵌入大小的方法往往导致性能降低或计算成本高.
研究的目的:
- 开发一个有效的框架,用于在推模型中分配定制嵌入大小.
- 为了减少内存使用和嵌入层中的参数数量,而不会影响模型性能.
主要方法:
- 提出了一个基于修剪的多尺寸嵌入 (PME) 框架,从修剪的角度来看,该框架从修剪的角度来看尺寸分配.
- 在搜索阶段对模型性能影响最小的 Prunes 嵌入尺寸.
- 转移修剪嵌入的容量,以获得定制的令牌大小,降低搜索成本.
主要成果:
- PME框架有效地识别了类别特征的合适嵌入大小.
- 实现强大的推模型性能,同时显著减少参数数量.
- 与传统的固定大小嵌入方法相比,显示了大量的内存节省.
结论:
- PME框架为优化推系统中嵌入大小提供了有效和计算效率高的解决方案.
- 能够显著减少参数和节省内存,使推模型更具可扩展性.
相关概念视频
Survival Tree
117
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
117
Downsampling
192
When considering a sampled sequence with zero values between sampling instants, one can replace it by taking every N-th value of the sequence. At these integer multiples of N, the original and sampled sequences coincide. This process, known as decimation, involves extracting every N-th sample from a sequence, thereby creating a more efficient sequence.
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
The Fourier transform of the decimated sequence reveals a combination of scaled and shifted versions of the original spectrum. This...
192
Upsampling
265
Managing signal sampling rates is essential in digital signal processing to maintain signal integrity. A decimated signal, characterized by a reduced frequency range due to its lower sampling rate, can be upsampled by inserting zeros between each sample. This upsampling process expands the original spectrum and introduces repeated spectral replicas at intervals dictated by the new Nyquist frequency. To refine this zero-inserted sequence, it is passed through a lowpass filter with a cutoff...
265
Sample Size Calculation
3.6K
Knowledge of the sample size is the first requirement to conduct random sampling or an experiment. The sample size is the total number of units, observations, or groups (in some cases) used to get the data to estimate a population parameter. As the name suggests, the sample size is that of the sample drawn from the population and differs from the population size.
The sample size for the given experiment or sampling effort is fundamental to any study design. Sample size decides the number of...
The sample size for the given experiment or sampling effort is fundamental to any study design. Sample size decides the number of...
3.6K
Quantifying and Rejecting Outliers: The Grubbs Test
1.7K
Sometimes, a data set can have a recorded numerical observation that greatly deviates from the rest of the data. Assuming that the data is normally distributed, a statistical method called the Grubbs test can be used to determine whether the observation is truly an outlier. To perform a two-tailed Grubbs test, first, calculate the absolute difference between the outlier and the mean. Then, calculate the ratio between this difference and the standard deviation of the sample. This...
1.7K
Maximum Size of Aggregate
168
The maximum size of aggregate is defined as the aperture of the sieve retaining 15 percent or more of the particles present in the aggregate sample. The aggregate's maximum size impacts the concrete's water requirement, workability, and strength. Larger aggregates reduce the surface area needing cement paste coverage, which can lower water needs, thereby allowing a decrease in the water-to-cement ratio when the desired workability and richness of the mix are to be maintained, which can...
168


