Weighting Methods for Rare Event Identification From Imbalanced Datasets.

Area of Science:

• Machine Learning
• Data Science
• Artificial Intelligence

Background:

• Machine learning models struggle with imbalanced datasets where rare events are crucial.
• Classifiers are often biased towards majority classes, hindering accurate rare event detection.
• Existing methods for imbalanced data, like sampling and basic weighting, have limitations.

Purpose of the Study:

• To develop novel weighting algorithms for effective rare event detection in machine learning.
• To address the challenge of identifying infrequent but important events in large datasets.
• To improve the performance of classifiers on imbalanced data, particularly in real-time applications.

Main Methods:

• Proposed a boosting-style algorithm for computing class weights with theoretical guarantees.
• Developed an adaptive weighting algorithm suitable for real-time network monitoring and similar applications.
• Focused on the weighting approach to rebalance class importance without discarding data.

Main Results:

• The proposed algorithms demonstrated superior performance compared to existing weighting and boosting methods.
• Effectiveness was validated on power grid data and various public imbalanced datasets.
• The algorithms showed enhanced superiority when dealing with noisy data, improving rare event identification.

Conclusions:

• The novel weighting algorithms offer a robust solution for rare event detection in imbalanced datasets.
• The adaptive nature allows for controlled trade-offs between detection rates and false alarms.
• These methods provide a significant advancement for applications like network monitoring and anomaly detection.