A Rigorous Behavior Assessment of CNNs Using a Data-Domain Sampling Regime.

Area of Science:

• Computer Vision
• Data Visualization
• Machine Learning

Background:

• Convolutional Neural Networks (CNNs) are increasingly used for image analysis.
• Understanding CNNs' perception of visual data, like charts, is crucial.
• Current methods for evaluating CNN graphic perception are limited.

Purpose of the Study:

• To introduce a novel data-domain sampling regime for quantifying CNN graphic perception.
• To assess CNNs' ratio estimation abilities in bar charts.
• To compare CNN performance against human observers.

Main Methods:

• Developed a data-domain sampling regime for evaluating CNNs.
• Analyzed 16 million trials from 800 CNN models and 6,825 trials from 113 human participants.
• Assessed CNNs on sensitivity to distribution discrepancies, sample stability, and human-like expertise.

Main Results:

• CNNs can outperform human observers in bar chart ratio estimation.
• CNN biases are directly correlated with the training-test data distribution distance.
• CNNs exhibit predictable and elegant behaviors when interpreting visualizations.

Conclusions:

• CNNs possess a robust graphic perception capability.
• The training-test distribution distance is a key factor influencing CNN performance and biases.
• The developed regime provides actionable insights into CNN visual interpretation.