F-DATA:Fugaku工作负载数据集,用于HPC系统中以工作为中心的预测建模
Francesco Antici1, Andrea Bartolini2, Jens Domke3
1University of Bologna, Department of Electrical, Electronic, and Information Engineering Guglielmo Marconi, Bologna, 40121, Italy. francesco.antici@unibo.it.
Scientific data
|July 30, 2025
概括
本研究介绍了F-DATA,这是来自Fugaku超级计算机的大型公共数据集,用于帮助预测高性能计算 (HPC) 工作特征. 这个资源可以更好地管理工作负载,以优化吞吐量和减少对环境的影响.
科学领域:
- 高性能计算 (HPC) 是一种高性能计算.
- 可持续的计算 可持续的计算
- 数据科学数据科学数据科学
背景情况:
- 高性能计算 (HPC) 系统加速科学发现,但对环境产生重大影响 (碳足迹,能源消耗).
- 在最大限度地减少环境影响的同时,优化HPC系统的吞吐量需要准确地预测工作执行特征.
- 由于大型公共HPC数据集的稀缺性,开发预测模型具有挑战性.
研究的目的:
- 引入F-DATA,这是一个全面的公开数据集,包含大约2400万个在Fugaku超级计算机上执行的任务.
- 为开发和验证预测HPC工作执行特征的模型提供丰富的功能集.
- 促进对可持续HPC开发工作负载管理策略的研究.
主要方法:
- 从Fugaku超级计算机上执行的2400万个工作中编制的数据集.
- 包括匿名和不可逆转地编码的敏感工作数据.
- 应用自然语言处理 (NLP) 模型来编码敏感数据,同时保持预测效用.
主要成果:
- 创建F-DATA,一个大规模的公共数据集,用于HPC工作分析.
- 该数据集包含了广泛的功能,适用于各种预测任务.
- 通过基于NLP的编码来保存敏感的工作信息,解决隐私问题.
结论:
- F-DATA解决了在HPC研究中对公共数据集的关键需求.
- 该数据集支持开发先进的工作负载管理,以实现高效和可持续的超级计算.
- 保护隐私的数据编码可以更广泛地使用敏感的HPC执行信息.
相关概念视频
01:20Maxwell-Boltzmann Distribution: Problem Solving
1.7K
Individual molecules in a gas move in random directions, but a gas containing numerous molecules has a predictable distribution of molecular speeds, which is known as the Maxwell-Boltzmann distribution, f(v).
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
1.7K
02:30Quantifying Work
21.2K
As a system undergoes a change, its internal energy can change, and energy can be transferred from the system to the surroundings, or from the surroundings to the system.
21.2K
01:19Distributed Loads
613
Distributed loads are a common type of load that engineers and scientists encounter in various practical situations. Distributed loads often refer to a type of load spread over a surface or a structure and can be modeled as continuous force per unit area.
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
613
01:20Parallel Processing
229
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
229
01:21Distributed Loads: Problem Solving
738
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
738
01:10Work and Energy for Variable Forces
3.9K
When an object is acted upon by a variable force, the amount of work done and the change in energy of the object can be more complex to calculate compared to when a constant force is applied. Work is the product of force and displacement, while energy is the capacity of a system to do work. When a constant force is applied to an object, the work done can be calculated as the product of the force and the distance moved in the direction of the force. However, when a variable force is applied, the...
3.9K