Material fictions: Comparing physically based renderings and generative AI images through material perception

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Perceptual Psychology

Background:

• Generative artificial intelligence (AI) models are creating novel image types.
• These AI images can be visually convincing but not physically plausible.
• Investigating AI images' role in visual perception and as a new medium is crucial.

Purpose of the Study:

• To determine if generative AI images align with a medium-independent perceptual space.
• To compare human similarity judgments of AI-generated images with physically based renderings (PBR).
• To assess the structural consistency of AI image generation across different models and constraints.

Main Methods:

• Human similarity judgments were used to compare images from three AI models (DALL-E 2, Midjourney v2, Stable Diffusion v1.5) against a BRDF dataset (MERL).
• Experiment 1: AI models generated images from text prompts of 32 materials.
• Experiment 2: Stable Diffusion used depth-map constraints with the same text prompts.

Main Results:

• Initial AI models (DALL-E 2, Midjourney v2) produced unrelated 2D perceptual spaces compared to the 1D MERL space.
• Stable Diffusion with depth-map constraints generated robust, highly similar 2D perceptual spaces.
• These constrained AI spaces showed structural similarity to MERL and other material perception studies.

Conclusions:

• Generative AI images, particularly when constrained, can form a new medium for exploring material perception.
• AI models demonstrate potential for creating perceptually relevant imagery.
• Further research can leverage AI for understanding visual perception across different image modalities.