分布式计算工作流程的数据管理:基于iRODS的设置及其性能
Mohamad Hayek1, Martin Golasowski2, Stephan Hachinger1
1Leibniz Supercomputing Centre (LRZ), Bavarian Academy of Sciences and Humanities, Garching near Munich, Germany.
PloS one
|January 12, 2026
概括
像iRODS和B2SAFE这样的数据管理框架可以有效地处理分布式计算工作流. 优化的配置允许数据传输和网络带宽,证明它们适用于联合基础设施的适用性.
科学领域:
- 科学数据管理科学数据管理
- 高性能计算基础设施的基础设施.
- 分布式数据系统 分布式数据系统
背景情况:
- 现代数据管理框架在日常使用中被评估其实际效率.
- 框架必须支持地理分布的高性能和云计算系统.
- 数据传输性能对于在分布式工作流中和网络带宽至关重要.
研究的目的:
- 测试数据管理框架对于分布式计算工作流程的适用性.
- 评估iRODS与B2SAFE作为LEXIS平台的数据后端的性能.
- 测量超级计算站点之间的广域网数据传输性能.
主要方法:
- 与EUDAT的B2SAFE模块集成的规则导向数据系统 (iRODS) 的评估.
- 在LEXIS平台内构建和评估数据基础设施.
- 测量超级计算站点之间在广域网上的数据传输性能.
主要成果:
- 通过优化客户端配置和文件大小,可以高效地利用网络带宽.
- iRODS适合集成到联合计算基础设施中,支持OpenID Connect和在线服务.
- 确定了局限性,并发现了对数据传输性能的优化机会.
结论:
- 像iRODS这样的数据管理系统能够满足联合计算基础设施的需求.
- 该研究表明了分布式环境中高性能数据传输的潜力.
- 计划继续利用这些属性,以优化像EXA4MIND这样的项目中的数据密集的工作流.
更多相关视频
12:10An Experimental and Bioinformatics Protocol for RNA-seq Analyses of Photoperiodic Diapause in the Asian Tiger Mosquito, Aedes albopictus
Published on: November 30, 2014
13.8K
09:49Routine Collection of High-Resolution cryo-EM Datasets Using 200 KV Transmission Electron Microscope
Published on: March 16, 2022
5.9K
相关概念视频
Distributed Loads
939
Distributed loads are a common type of load that engineers and scientists encounter in various practical situations. Distributed loads often refer to a type of load spread over a surface or a structure and can be modeled as continuous force per unit area.
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
For example, consider a bookshelf filled with books stacked vertically adjacent to each other. The weight of the books is evenly distributed over the length of the shelf. As a result, the pressure at different locations on the surface of the...
939
Distributed Loads: Problem Solving
1.1K
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
1.1K
Parallel Processing
626
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
626
Distribution Reliability and Automation
489
Distribution reliability in electrical power systems is critical for ensuring an uninterrupted power supply to consumers at minimal cost. According to IEEE Standard Terms, reliability is the probability that a device will function without failure over a specified time period or amount of usage. For electric power distribution, this translates to maintaining continuous power supply and addressing customer concerns over power outages. Several indices, as defined by IEEE Standard 1366-2012, are...
489
Quantifying Work
23.9K
As a system undergoes a change, its internal energy can change, and energy can be transferred from the system to the surroundings, or from the surroundings to the system.
23.9K
Distribution of Molecular Speeds
5.3K
The motion of molecules in a gas is random in magnitude and direction for individual molecules, but a gas of many molecules has a predictable distribution of molecular speeds. This predictable distribution of molecular speeds is known as the Maxwell-Boltzmann distribution. The distribution of molecular speeds in liquids is comparable to that of gases but not identical and can help to understand the phenomenon of the boiling and vapor pressure of a liquid. Consider that a molecule requires a...
5.3K
