Learning Numerosity Representations with Transformers: Number Generation Tasks and Out-of-Distribution Generalization

Area of Science:

• Artificial Intelligence
• Deep Learning
• Numerical Cognition

Background:

• Deep learning models aim to disentangle data variations, often using conditional models due to the difficulty of modeling joint probability mass functions.
• In numerical cognition, deep learning has shown promise in emergent numerosity representations from images.
• Existing models primarily estimate numerosity from images, not generate images based on a given number.

Purpose of the Study:

• To develop deep learning models capable of conditionally generating synthetic images with a specified number of items.
• To address the more challenging task of image generation based on numerical input, moving beyond estimation.

Main Methods:

• Utilized attention-based deep learning architectures operating at the pixel level.
• Trained models on tasks requiring the generation of images with a target number of items.

Main Results:

• Demonstrated that attention-based architectures can learn to generate visually coherent images.
• Showed successful generation of images with approximately the target number of items, including novel numerosities not present in training data.

Conclusions:

• Attention-based models show potential for generating images with specific numerosities, advancing generative capabilities in numerical cognition.
• This approach extends deep learning applications in numerical cognition to generative tasks, offering new avenues for research.