Automatic Detection of Previously-Unseen Application States for Deployment Environment Testing and Analysis

Area of Science:

• Software Engineering
• Computer Science
• Systems and Software Engineering

Background:

• Testing large, complex software systems exhaustively in all states is infeasible due to time and cost constraints.
• Post-deployment testing and analysis in the field are common but often limited by high performance overhead from instrumentation.
• Existing automated approaches struggle with redundant tests and analyses in already explored application states.

Purpose of the Study:

• To develop a technique that ensures deployment environment tests are executed only in previously unencountered application states.
• To reduce the performance overhead associated with automated testing and analysis of deployed software.
• To enhance the efficiency and effectiveness of fault detection, model checking, security testing, and fault localization in production environments.

Main Methods:

• Implementation of a solution to track and execute tests exclusively in novel application states.
• Empirical study to evaluate the effectiveness of the proposed approach.
• Generalization of the technique for broader application in automated testing and analysis.

Main Results:

• Demonstrated effectiveness of the approach in empirical studies.
• Significant reduction in redundant tests and analyses by focusing on unexplored states.
• Successful implementation of a system that prioritizes testing in new application states.

Conclusions:

• The proposed method effectively reduces testing overhead for complex software systems in deployment environments.
• This technique can be generalized to improve various automated testing and analysis methods.
• Prioritizing tests in novel states offers a practical solution to the limitations of traditional post-deployment software analysis.