A Systematic Approach for Evaluating Artificial Intelligence Models in Industrial Settings.

Area of Science:

• Machine Learning
• Deep Learning
• Data Science

Background:

• Artificial Intelligence (AI) is crucial for data analysis in industry's digital transformation.
• Machine Learning (ML) models often underperform in real-world conditions due to data perturbations.
• Model robustness against unexpected or degraded data is a significant challenge in AI deployment.

Purpose of the Study:

• To investigate the robustness of seven Machine Learning (ML) and Deep Learning (DL) algorithms.
• To evaluate algorithm performance in classifying univariate time-series data under induced perturbations.
• To assess the predictability of model robustness using decision trees for various data quality issues.

Main Methods:

• A systematic approach was developed to artificially inject two common data collection perturbations.
• Seven ML/DL algorithms were tested on twenty univariate time-series datasets from the UCR repository.
• Model robustness was evaluated based on performance degradation under perturbed data conditions.

Main Results:

• Significant disparities in model robustness were observed across the evaluated algorithms.
• Data quality degradation severely impacted the performance of some ML and DL models.
• Predicting the impact of robustness using decision trees proved not straightforward.

Conclusions:

• No single ML or DL algorithm demonstrated consistent robustness across all tested perturbations.
• A systematic evaluation approach is essential for assessing AI model robustness in real-world scenarios.
• Further research is needed to develop reliable methods for predicting model robustness.