Prioritizing test cases for deep learning-based video classifiers

Area of Science:

• Software Engineering
• Machine Learning
• Computer Vision

Background:

• Video applications are prevalent, but video test case labeling is expensive due to temporal data and large volumes.
• Existing test prioritization methods fail to leverage the temporal information unique to video data.
• Efficiently assessing the accuracy of video-based systems requires addressing labeling costs and temporal complexities.

Purpose of the Study:

• To introduce VRank, the first test prioritization approach specifically designed for video test inputs.
• To reduce the cost and effort associated with labeling video test cases for system accuracy assessment.
• To improve the efficiency of identifying misclassified video test cases, thereby detecting system faults earlier.

Main Methods:

• Developed VRank, a novel test prioritization technique tailored for video data.
• Trained a ranking model to predict the misclassification probability of video test inputs by a deep neural network (DNN) classifier.
• Utilized four feature types for prediction: temporal features (TF), video embedding features (EF), prediction features (PF), and uncertainty features (UF).

Main Results:

• VRank effectively prioritizes video test cases based on their predicted misclassification probabilities.
• Empirical evaluation with 120 subjects demonstrated VRank's superior performance over existing methods.
• VRank achieved significant average improvement: 5.76%–46.51% on natural datasets and 4.26%–53.56% on noisy datasets.

Conclusions:

• VRank is a highly effective test prioritization approach for video inputs, outperforming traditional methods.
• The method successfully addresses the unique challenges posed by video data in test case prioritization.
• VRank offers a practical solution for reducing labeling costs and accelerating fault detection in video-based systems.