Testing convolutional neural network based deep learning systems: a statistical metamorphic approach.

Area of Science:

• Artificial Intelligence
• Software Engineering
• Machine Learning

Background:

• Machine learning (ML) is crucial in domains like healthcare and autonomous driving.
• Metamorphic testing (MT) is effective for complex systems lacking or with difficult oracles.
• Conventional MT struggles with the stochastic nature of deep learning models, like convolutional neural networks (CNNs).

Purpose of the Study:

• To address the limitations of conventional MT in verifying stochastic deep learning models.
• To introduce a statistical metamorphic testing (SMT) technique that bypasses the need for fixed random seeds.
• To propose an algorithm for minimizing metamorphic relations (MRs) to optimize testing resources.

Main Methods:

• Proposed seven novel MRs tailored for deep learning models.
• Integrated statistical methods to verify adherence to MRs without deterministic outputs.
• Employed mutation testing to demonstrate the approach's effectiveness on CNNs for pneumonia detection.
• Developed an MR minimization algorithm for computational efficiency.

Main Results:

• The SMT approach successfully uncovered 85.71% of implementation faults in the classifiers under test (CUT).
• The proposed MRs and statistical verification methods effectively validated deep learning models.
• The MR minimization algorithm demonstrated potential for significant savings in testing resources.

Conclusions:

• SMT offers a robust solution for testing stochastic deep learning models, overcoming limitations of traditional MT.
• The proposed technique enhances fault detection rates in critical AI applications, particularly in healthcare.
• The MR minimization algorithm contributes to more efficient and cost-effective software testing practices.