An information theoretic score for learning hierarchical concepts

Area of Science:

• Cognitive Science
• Machine Learning
• Computational Neuroscience

Background:

• Human learning robustly extracts regularities from complex, noisy environments, often through unsupervised interaction.
• Hierarchical structures in the world and brain facilitate efficient knowledge representation and symbolic processing.
• Understanding the drivers of hierarchical concept acquisition is crucial for artificial intelligence and cognitive modeling.

Purpose of the Study:

• To investigate the role of prediction advancement as a driver for learning hierarchical spatiotemporal concepts.
• To introduce and evaluate an information-theoretic score (CORE) for guiding unsupervised concept formation.
• To develop an integrated learning system capable of building complex concepts from primitive elements.

Main Methods:

• Developed a 'prediction games' framework where concepts act as predictors, targets, and building blocks.
• Implemented a system that learns hierarchical concepts from raw text, starting with characters.
• Introduced the CORE score, comparing prediction performance against a baseline and balancing prediction strength with observational accuracy.

Main Results:

• The CORE score effectively motivates the construction of larger, more complex concepts.
• The learning system demonstrates scalability, forming thousands of concepts from hundreds of thousands of episodes.
• Empirical comparisons show distinct advantages and differences compared to transformer neural networks and n-gram models.

Conclusions:

• Optimizing prediction is a key driver for unsupervised hierarchical concept learning.
• The CORE score provides a scalable and open-ended mechanism for concept formation.
• This approach offers a promising foundation for more sophisticated concept structures and advanced AI systems.