Machine vision benefits from human contextual expectations.

Area of Science:

• Computer Vision
• Cognitive Science
• Artificial Intelligence

Background:

• Scene context aids object recognition in both humans and machines.
• Human and machine training experiences differ, potentially leading to distinct context representations.
• Humans often experience scenes devoid of expected objects, unlike typical machine training data.

Purpose of the Study:

• To develop a method for measuring human contextual expectations.
• To investigate if human-derived contextual expectations can improve machine vision algorithms.
• To determine if human and machine context representations are qualitatively different.

Main Methods:

• Developed a paradigm to measure human expectations of object scale, location, and likelihood in object-absent scenes.
• Used scene features to predict human expectations on novel scenes.
• Augmented deep neural network (DNN) decisions with predicted human expectations.

Main Results:

• Human contextual expectations were systematic and predictable from scene features.
• Augmenting DNNs with human expectations significantly improved object detection accuracy (1-3%).
• Associated object detection saw even larger gains (3-20%) compared to conventional computer vision features.

Conclusions:

• Human-derived contextual expectations demonstrably improve deep neural network performance.
• The findings suggest qualitative differences between human and deep neural network context representations.
• Integrating human cognitive insights offers a promising avenue for advancing machine vision.