Risks of ignoring uncertainty propagation in AI-augmented security pipelines

Area of Science:

• Computer Science
• Artificial Intelligence
• Software Engineering

Background:

• AI is increasingly integrated into software development, creating automated pipelines with AI subsystems of uncertain performance.
• This integration poses significant risks to safety-critical domains due to potential error propagation.
• Existing risk analysis methods do not adequately address the uncertainty in AI-augmented systems.

Purpose of the Study:

• To develop a formal framework for capturing and quantifying uncertainty propagation in AI-augmented software systems.
• To create a simulator for evaluating the impact of propagating errors.
• To provide recommendations for AI system evaluation policies.

Main Methods:

• Formalizing the underpinnings of uncertainty propagation in AI pipelines.
• Developing a simulator to quantify the uncertainty arising from error propagation.
• Conducting a case study to evaluate the simulation of propagating errors.

Main Results:

• The study provides a method to formally capture uncertainty propagation in AI pipelines.
• A simulator was developed and evaluated, demonstrating the quantification of uncertainty.
• The approach's generalizability and limitations were discussed, with recommendations for evaluation policies.

Conclusions:

• The developed framework and simulator offer a novel approach to assessing uncertainty in AI-augmented systems.
• The findings are crucial for enhancing the safety and reliability of AI in critical applications.
• Further research is needed to extend the approach to real-world systems and relax existing assumptions.