关于预训练数据的档案视角
Meera A Desai1, Irene V Pasquetto2, Abigail Z Jacobs1
1School of Information, University of Michigan, Ann Arbor, MI, USA.
Patterns (New York, N.Y.)
|April 22, 2024
概括
大型语言模型 (LLM) 预训练数据集被分析为非正式的档案. 这种档案镜头揭示了主观的价值决策和超出模型行为范围的更广泛的社会影响.
科学领域:
- 计算机科学 计算机科学
- 信息科学 信息科学 信息科学
- 档案研究 档案研究
背景情况:
- 大型语言模型 (LLM) 的快速发展导致了大量预训练数据集的创建,这些数据集通常来自网络.
- 这些数据集对于LLM开发至关重要,但它们的创建和构成引发了重大问题.
研究的目的:
- 用档案研究的框架分析预训练数据集.
- 识别超越直接模型性能的预训练数据创建和使用的影响.
- 为了突出数据集策划中固有的主观价值判断.
主要方法:
- 应用档案研究原则,将预培训数据集概念化为非正式档案.
- 检查这些数据集的异质性质及其在调解知识获取中的作用.
- 调查数据集构建中包含和排除决策的影响.
主要成果:
- 预培训数据集充当非正式档案的功能,调解获取多样化的知识.
- 策划这些数据集的过程涉及对价值观和优先事项的主观决定.
- 除了塑造模型行为之外,数据集的创建还有重要的社会和伦理影响.
结论:
- 档案视角提供了一种新的方式来理解LLM预培训数据的复杂性.
- 承认数据集是非正式的档案照亮了主观的选择和价值体系嵌入其中.
- 该框架为研究人员提供了解决大规模数据创建和其社会影响方面的挑战的机会.
相关概念视频
Archival Research
16.0K
Some researchers gain access to large amounts of data without interacting with a single research participant. Instead, they use existing records to answer various research questions. This type of research approach is known as archival research. Archival research relies on looking at past records or data sets to look for interesting patterns or relationships. For example, a researcher might access the academic records of all individuals who enrolled in college within the past ten years and...
16.0K
Survival Tree
84
Survival trees are a non-parametric method used in survival analysis to model the relationship between a set of covariates and the time until an event of interest occurs, often referred to as the "time-to-event" or "survival time." This method is particularly useful when dealing with censored data, where the event has not occurred for some individuals by the end of the study period, or when the exact time of the event is unknown.
Building a Survival Tree
Constructing a...
Building a Survival Tree
Constructing a...
84
Elaborative Rehearsals
86
Elaborative rehearsal is a crucial cognitive strategy that strengthens information encoding in long-term memory by making meaningful connections between new data and pre-existing knowledge. This approach contrasts with maintenance rehearsal, which involves simple repetition without delving into the significance of the information. While maintenance rehearsal might temporarily keep information active in short-term memory, it is less effective for long-term retention.
The effectiveness of...
The effectiveness of...
86
Associative Learning
350
Associative learning is a fundamental concept in behavioral psychology, wherein a connection is established between two stimuli or events, leading to a learned response. This process is critical in understanding how behaviors are acquired and modified. Conditioning, the mechanism through which associations are formed, can be divided into two main types: classical conditioning and operant conditioning, each elucidating different aspects of associative learning.
Classical conditioning, also known...
Classical conditioning, also known...
350
Improving Translational Accuracy
10.3K
Base complementarity between the three base pairs of mRNA codon and the tRNA anticodon is not a failsafe mechanism. Inaccuracies can range from a single mismatch to no correct base pairing at all. The free energy difference between the correct and nearly correct base pairs can be as small as 3 kcal/ mol. With complementarity being the only proofreading step, the estimated error frequency would be one wrong amino acid in every 100 amino acids incorporated. However, error frequencies observed in...
10.3K


