Leveraging History to Predict Infrequent Abnormal Transfers in Distributed Workflows.

Area of Science:

• Scientific computing
• Network performance analysis
• Machine learning applications

Background:

• Distributed data-intensive applications in scientific computing depend on efficient data sharing.
• Slow network connections create significant bottlenecks in distributed workflows, impacting research progress.
• Identifying these slow connections is challenging due to their low frequency in well-maintained networks.

Purpose of the Study:

• To develop a predictive model for identifying slow network connections in scientific computing environments.
• To address the class-imbalance problem inherent in detecting rare events like slow data transfers.
• To evaluate the effectiveness of stratified sampling techniques in improving machine learning model performance.

Main Methods:

• Analysis of network traffic logs from January 2021 to August 2022 at the National Energy Research Scientific Computing Center (NERSC).
• Feature engineering based on historical network traffic patterns to identify low-performing data transfers.
• Application and comparison of various stratified sampling techniques to mitigate class imbalance.
• Training and evaluation of machine learning models using balanced datasets.

Main Results:

• A set of history-based features was defined for detecting slow data transfers.
• Stratified sampling, specifically undersampling normal connections, proved highly effective for model training.
• The developed model achieved a high F1 score of 0.926 in predicting slow connections.

Conclusions:

• Predicting slow network connections in scientific computing is feasible and crucial for workflow optimization.
• Undersampling normal data is a simple yet powerful technique to overcome class imbalance in this domain.
• The findings offer a practical approach to enhance the reliability and efficiency of distributed scientific computing.