Comparison of Anomaly Detectors: Context Matters

Area of Science:

• Artificial Intelligence
• Machine Learning
• Data Science

Background:

• Deep generative models are increasingly used for anomaly detection.
• Published results on these methods often show variability and contradictions.
• Classical methods are being challenged by these new approaches.

Purpose of the Study:

• To compare anomaly detection methods, focusing on deep generative models.
• To identify sources of variability in experimental results.
• To provide a clearer understanding of method performance across different conditions.

Main Methods:

• Comparative analysis of various anomaly detection paradigms.
• Evaluation on diverse tabular and image datasets.
• Investigation of experimental conditions influencing results.

Main Results:

• Performance variability is linked to dataset type (tabular/image) and anomaly nature (statistical/semantic).
• Hyperparameter selection strategy, especially the number of anomalies in validation sets, significantly impacts results.
• Method performance differs across various experimental contexts and computational times.

Conclusions:

• The identified sources of variability explain discrepancies in prior research.
• Emphasizes the critical need for precise context specification when publishing new anomaly detection methods.
• Open-source code and results are provided for reproducibility and further research.