Aligning machine and human visual representations across abstraction levels

Area of Science:

• Artificial Intelligence
• Cognitive Science
• Computer Vision

Background:

• Deep neural networks (DNNs) are increasingly used as models for human behavior and neural representations.
• However, significant differences exist between DNN training and human learning, leading to poor generalization in models.
• Current vision models often fail to capture the hierarchical organization of human conceptual knowledge.

Purpose of the Study:

• To identify and address the misalignment between human conceptual knowledge and DNN representations.
• To develop more human-aligned vision models that exhibit improved generalization and robustness.
• To investigate methods for infusing human knowledge into artificial intelligence (AI) systems.

Main Methods:

• Trained a teacher model to imitate human judgments on conceptual similarity.
• Transferred human-aligned representational structure from the teacher model to state-of-the-art vision foundation models via fine-tuning.
• Evaluated models on similarity tasks using human judgments across multiple semantic abstraction levels.

Main Results:

• The human-aligned models demonstrated more accurate approximations of human behavior and uncertainty.
• Performance improved across diverse machine learning tasks, showing enhanced generalization and out-of-distribution robustness.
• Models successfully captured hierarchical semantic abstractions present in human cognition.

Conclusions:

• Integrating human knowledge into DNNs creates a hybrid representation that aligns better with human cognitive judgments.
• This approach leads to more robust, interpretable, and human-aligned AI systems.
• Infusing AI with human knowledge offers a promising path towards more capable and trustworthy artificial intelligence.