DPDispatcher:可扩展的HPC任务调度用于人工智能驱动的科学
Fengbo Yuan1, Zhaohan Ding2, Yun-Pei Liu3,4
1Department of Physics, University of Alabama at Birmingham, Birmingham, Alabama 35205, United States.
Journal of chemical information and modeling
|November 3, 2025
概括
DPDispatcher 是一个开源的Python框架,用于在高性能计算 (HPC) 环境中的任务调度. 它通过提供可扩展,容错的调度,减少错误和改善科学计算的自动化来增强人工智能驱动的工作流.
科学领域:
- 计算科学 计算科学
- 人工智能的人工智能
- 高性能计算 高性能计算
背景情况:
- 人工智能正在改变计算科学,但人工智能驱动的工作流程很复杂,跨越多种HPC系统.
- 现有的工作流面临着跨异质计算环境的可扩展性,容错性和自动化方面的挑战.
研究的目的:
- 介绍DPDispatcher,这是一个开源的Python框架,用于可扩展和容错的任务调度.
- 解决需要在人工智能驱动的计算科学工作流程中强大的任务管理的需求.
- 提高HPC系统上的科学计算任务的可移植性和自动化.
主要方法:
- 开发了DPDispatcher作为一个Python框架,强调轻量级提交,自动重试和强大的恢复.
- 独立连接和文件分阶段从调度器控制的模块化.
- 支持多个HPC任务管理器,并提供本地和安全 (SSH) 后端.
主要成果:
- DPDispatcher已经被十多个科学包所采用.
- 在主动学习,自由能量计算,材料选和LLM驱动的HPC代理中展示了使用案例.
- 减少科学计算工作流程中的运营开销和错误率.
结论:
- DPDispatcher 提高了高通量科学计算的可靠性和自动化.
- 该框架提高了AI驱动的工作流在多种HPC系统中的可移植性.
- DPDispatcher促进了科学研究中复杂的计算任务的高效执行.
相关概念视频
01:20Parallel Processing
621
The brain processes sensory information rapidly due to parallel processing, which involves sending data across multiple neural pathways at the same time. This method allows the brain to manage various sensory qualities, such as shapes, colors, movements, and locations, all concurrently. For instance, when observing a forest landscape, the brain simultaneously processes the movement of leaves, the shapes of trees, the depth between them, and the various shades of green. This enables a quick and...
621
01:17Ampere-Maxwell's Law: Problem-Solving
1.1K
A parallel-plate capacitor with capacitance C, whose plates have area A and separation distance d, is connected to a resistor R and a battery of voltage V. The current starts to flow at t = 0. What is the displacement current between the capacitor plates at time t? From the properties of the capacitor, what is the corresponding real current?
To solve the problem, we can use the equations from the analysis of an RC circuit and Maxwell's version of Ampère's law.
For the first part of the...
To solve the problem, we can use the equations from the analysis of an RC circuit and Maxwell's version of Ampère's law.
For the first part of the...
1.1K
01:16Non-equilibrium in the Cell
5.3K
An important concept in studying metabolism and energy is that of chemical equilibrium. Most chemical reactions are reversible. They can proceed in both directions, releasing energy into their environment in one direction, and absorbing it from the environment in the other direction. The same is true for the chemical reactions involved in cell metabolism, such as the breaking down and building up of proteins into and from individual amino acids, respectively. Reactants within a closed system...
5.3K
01:20Maxwell-Boltzmann Distribution: Problem Solving
2.8K
Individual molecules in a gas move in random directions, but a gas containing numerous molecules has a predictable distribution of molecular speeds, which is known as the Maxwell-Boltzmann distribution, f(v).
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
This distribution function f(v) is defined by saying that the expected number N (v1,v2) of particles with speeds between v1 and v2 is given by
2.8K
01:21Distributed Loads: Problem Solving
1.1K
Beams are structural elements commonly employed in engineering applications requiring different load-carrying capacities. The first step in analyzing a beam under a distributed load is to simplify the problem by dividing the load into smaller regions, which allows one to consider each region separately and calculate the magnitude of the equivalent resultant load acting on each portion of the beam. The magnitude of the equivalent resultant load for each region can be determined by calculating...
1.1K
01:29Mechanistic Models: Compartment Models in Algorithms for Numerical Problem Solving
277
Mechanistic models play a crucial role in algorithms for numerical problem-solving, particularly in nonlinear mixed effects modeling (NMEM). These models aim to minimize specific objective functions by evaluating various parameter estimates, leading to the development of systematic algorithms. In some cases, linearization techniques approximate the model using linear equations.
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
In individual population analyses, different algorithms are employed, such as Cauchy's method, which uses a...
277