Generating metamers of human scene understanding

Area of Science:

• Computer Vision
• Human Perception
• Artificial Intelligence

Background:

• Human vision integrates low-resolution peripheral gist with high-resolution foveal information for scene understanding.
• Generating realistic scene images aligned with human perception is a complex challenge.

Purpose of the Study:

• Introduce MetamerGen, a novel tool for generating image metamers that align with latent human scene representations.
• Investigate the role of visual processing levels in human scene judgments using generated metamers.

Main Methods:

• Developed MetamerGen, a latent diffusion model using a dual-stream representation (DINOv2 tokens) for foveated scenes.
• Fused detailed fixated features with peripherally degraded features to capture scene context.
• Conducted a same-different behavioral experiment to evaluate perceptual alignment.

Main Results:

• MetamerGen successfully generates image metamers that align with human scene representations.
• Proof-of-concept analyses identified specific features influencing human judgments.
• High-level semantic alignment, especially with viewer-specific fixations, strongly predicts metamerism.

Conclusions:

• MetamerGen is a powerful tool for studying human scene understanding and generating perceptually aligned images.
• The study highlights the importance of semantic information and fixation patterns in visual scene perception.