适应性存储储存储策略用于Spark环境中的大量工作负载
Bohan Li1, Xin He1, Junyang Yu1
1School of Software, Henan University, Kaifeng, Henan Province, China.
PeerJ. Computer science
|December 16, 2024
概括
一个自适应性存储储备 (AMR) 策略有效地解决了Apache Spark中的重量工作负载的存储瓶. 这种方法通过优化任务并行性和内存配置,显著减少执行时间,提高计算效率.
科学领域:
- 计算机科学 计算机科学
- 大数据分析大数据分析
- 分布式系统 分布式系统
背景情况:
- 物联网 (IoT) 和工业2.0推动了对广泛数据计算的需求.
- 由于内存计算,Apache Spark 是一个关键的大数据平台,但它面临着沉重工作负载的内存瓶.
- 这些瓶导致了弹性分布式数据集 (RDD) 的驱逐,并降低了计算效率.
研究的目的:
- 提出一个自适应性内存保留 (AMR) 策略,以减轻Apache Spark中的内存瓶问题.
- 优化任务并行性和内存配置,以实现高效的计算处理.
主要方法:
- 模拟最佳的任务并行性,最大限度地减少阻断和正常任务完成之间的差异.
- 确定任务并行性的最佳内存,以创建高效的执行内存空间.
- 基于任务进度的动态调整 (压缩/扩展) 实现了自适应执行内存保留.
- 为不同的RDD类型选择了合适的存储位置,考虑到缓存位置成本和实时内存使用情况.
主要成果:
- 拟议的AMR策略有效地减轻了Spark.中的执行内存压力.
- 广泛的实验验证了AMR与现有解决方案相比的有效性.
- 在执行时间方面,AMR实现了大约46.8%的减少.
结论:
- 在Apache Spark中,AMR策略为管理内存密集型工作负载提供了强大的解决方案.
- 优化的内存保留和动态调整提高了Spark的计算效率和性能.
相关概念视频
Chunking and Rehearsal in Sensory Memory
155
Improving short-term memory can be achieved through techniques like chunking and rehearsal. Chunking involves organizing information into larger, more manageable units. This technique is particularly useful for information that exceeds the typical memory span of between five and nine items. For instance, logging into an online account with a password like "ta89vq0179gz" involves grouping letters and numbers into three chunks—ta89, vq01, and 79gz. It makes large amounts of...
155
Working Memory
133
Working memory refers to a combination of components, including short-term memory and attention, that allow an individual to hold information temporarily as we perform cognitive tasks. It is an essential cognitive function that enables the execution of complex tasks such as problem-solving, comprehension, and reasoning. Unlike short-term memory, which simply involves the storage of information for a brief period, working memory involves the active manipulation and processing of this...
133
Chunking
58
Chunking is a powerful cognitive technique that improves short-term memory retention by organizing information into smaller, more manageable units. The brain, limited by working memory capacity, can more easily process and store information when it is divided into "chunks" rather than presented as discrete, unrelated elements. Chunking is especially useful when dealing with large amounts of information, such as numerical sequences, words, or complex ideas.
The principle behind chunking...
The principle behind chunking...
58
The Availability Heuristic
5.9K
A heuristic is a general problem-solving framework (Tversky & Kahneman, 1974). You can think of these as mental shortcuts that are used to solve problems. Different types of heuristics are used in different types of situations, and the impulse to use a heuristic occurs when one of five conditions is met (Pratkanis, 1989):
5.9K
Interference and Decay
107
Forgetting is a complex cognitive phenomenon influenced by several factors, among which interference and decay are particularly prominent. These processes explain why individuals often struggle to retrieve specific information from memory, leading to lapses in recall that can be observed in everyday situations.
Interference occurs when competing memories hinder the retrieval of particular information. It can be classified into two types: proactive and retroactive interference. Proactive...
Interference occurs when competing memories hinder the retrieval of particular information. It can be classified into two types: proactive and retroactive interference. Proactive...
107
Retrieval
89
Retrieval is the process of getting information out of memory storage and back into conscious awareness. This ability is essential for daily tasks like brushing hair and teeth, driving to work, and performing job duties. Retrieval occurs in three ways: recall, recognition, and relearning.
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...
Recall involves accessing information without cues, such as during an essay test, where individuals must retrieve facts and concepts from memory unaided. Another example is remembering the name of a colleague...
89


