Fine-grained knowledge about manipulable objects is well-predicted by contrastive language image pre-training.

Area of Science:

• Cognitive Science
• Computer Vision
• Artificial Intelligence

Background:

• Object recognition involves distinguishing similar items, crucial for tasks like tool selection.
• Human object knowledge is organized by behavioral dimensions: vision, manipulation, and function.
• Investigating if deep learning can replicate these human-centric object properties is essential.

Purpose of the Study:

• To determine if deep learning models can approximate the fine-grained behavioral dimensions of human object knowledge.
• To compare the performance of multimodal networks against image-only networks in predicting these dimensions.

Main Methods:

• Utilized CLIP-ViT, a multimodal network trained on extensive image-text pairs.
• Evaluated the model's ability to predict behavioral dimensions relevant to object manipulation and function.
• Compared CLIP-ViT against other networks pre-trained on image-only datasets.

Main Results:

• Behavioral dimensions of object knowledge were generally well-predicted by CLIP-ViT.
• CLIP-ViT demonstrated superior performance compared to networks trained solely on images.
• The findings highlight the model's capacity for approximating nuanced object understanding.

Conclusions:

• CLIP-ViT effectively approximates fine-grained object knowledge, demonstrating the power of multimodal learning.
• Multimodal pre-training and large-scale datasets significantly benefit the model's ability to capture behavioral dimensions.
• This research bridges cognitive science and AI, offering insights into artificial object recognition capabilities.