Learning Assurance Analysis for Further Certification Process of Machine Learning Techniques: Case-Study Air Traffic...

Area of Science:

• Aviation Safety
• Artificial Intelligence
• Machine Learning

Background:

• Designing trustworthy artificial intelligence (AI) systems is a key challenge for aviation.
• The European Union Aviation Safety Agency (EASA) has proposed a user guide for AI regulation.
• Understanding the learning assurance process is vital for building trust in machine learning (ML) models.

Purpose of the Study:

• To analyze the validity and adaptability of the EASA W-shaped methodology for ML-based systems in air traffic control.
• To investigate the impact of the learning assurance process on trust in ML models for aviation.
• To address the time-dependent nature of ML algorithms in safety-critical applications.

Main Methods:

• Development of an ML-based conflict detection tool for identifying aircraft separation infringements.
• Utilizing extreme gradient boosting for classification and regression tasks.
• Evaluating the EASA W-shaped methodology in the context of time-dependent ML analysis.

Main Results:

• The EASA W-shaped methodology lacks specific provisions for time-dependent analysis in ML systems.
• Time significantly impacts the performance of ML algorithms when temporal requirements are not considered.
• Standard classification and regression metrics are insufficient for systems where prediction timing is critical.

Conclusions:

• The EASA W-shaped methodology requires enhancement to incorporate time-dependent factors for ML systems.
• Aviation safety regulations for AI must account for the dynamic nature of ML model predictions.
• Metrics used to evaluate ML models in time-sensitive aviation applications need to be time-aware.