Ensemble Prediction of Job Resources to Improve System Performance for Slurm-Based HPC Systems.

Area of Science:

• High-Performance Computing (HPC)
• Machine Learning Applications
• Resource Management

Background:

• HPC systems require efficient resource allocation for optimal performance.
• Predicting job resource demands (memory, time) is crucial for administrators and users.
• Current methods may not accurately forecast resource needs, leading to underutilization or delays.

Purpose of the Study:

• To develop and validate a novel machine learning methodology for predicting job resource requirements on HPC systems.
• To enhance the efficiency and utilization of HPC resources.
• To reduce power consumption in HPC environments.

Main Methods:

• Utilized historical job data (saccount data) from the Slurm workload manager.
• Applied supervised machine learning algorithms, including scikit-learn models and LightGBM.
• Trained and tested the model on over 200,000 jobs from two HPC providers (RMACC Summit and Beocat).

Main Results:

• Achieved up to 86% accuracy in predicting memory and time requirements for HPC jobs.
• Significantly reduced average waiting time (e.g., from 380 to 4 hours in RMACC Summit).
• Dramatically decreased turnaround time and achieved up to 100% HPC resource utilization.

Conclusions:

• The developed ML model effectively predicts job resource needs, benefiting HPC administrators and users.
• The methodology enhances HPC system efficiency, utilization, and potentially reduces energy consumption.
• This work represents a significant step towards an open-source tool for HPC resource management.