Predicting queue wait time probabilities for multi-scale computing

Area of Science:

• High Performance Computing (HPC)
• Computational Science
• Operations Research

Background:

• Supercomputing clusters face challenges in efficient resource allocation due to unpredictable job queue wait times.
• Current resource selection mechanisms often lack accurate estimations of job queuing duration.
• End-users require better tools to predict their job completion times for effective workflow management.

Purpose of the Study:

• To develop a robust method for predicting queue wait times in multi-site High Performance Computing environments.
• To integrate queue wait time predictions into multi-criteria brokering for optimized resource selection.
• To provide end-users with probabilistic estimates of job start and completion times.

Main Methods:

• Utilized historical system data for predictive modeling.
• Applied Bayes theorem with an independence assumption for probability calculations.
• Developed a REST API for service accessibility.
• Calibrated probability estimates for enhanced accuracy.

Main Results:

• The prediction system returns probability estimates (t_i, p_i) indicating the likelihood of a job starting before time t_i.
• Empirical evidence supports the efficacy of the developed prediction method.
• The probabilistic output offers more comprehensive information than traditional single-value regression estimates.

Conclusions:

• The proposed method enhances resource selection in HPC by incorporating queue wait time predictions.
• The system empowers end-users with valuable insights into their job scheduling and completion timelines.
• This approach offers a more informative alternative to standard queue wait time estimation techniques.