Multi-teacher knowledge distillation framework for lightweight anomaly detection

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Anomaly detection is crucial for system safety and security.
• Extreme class imbalance in datasets hinders traditional models' ability to identify anomalies.
• Efficient deployment of anomaly detection models is challenging.

Purpose of the Study:

• To develop a novel framework for anomaly detection that addresses extreme class imbalance.
• To integrate knowledge distillation with multiple resampling strategies for improved learning.
• To achieve model compression for efficient real-time deployment.

Main Methods:

• A multi-teacher knowledge distillation framework (MTKD) was proposed.
• Teacher models were trained on resampled datasets using diverse oversampling and undersampling techniques.
• A compact student model learned from multiple teachers, balancing normal and anomalous samples.

Main Results:

• The proposed method effectively addresses extreme class imbalance in anomaly detection.
• The compressed student model demonstrated enhanced generalization, reduced overfitting, and improved robustness.
• The framework achieved high accuracy, efficiency, and inference speed suitable for real-time applications.

Conclusions:

• The novel MTKD framework offers a robust solution for anomaly detection in imbalanced datasets.
• The approach is domain-agnostic and effective across various real-world scenarios like fraud and intrusion detection.
• This method provides a practical and efficient solution for real-time anomaly detection.