Fine-grained image generation with EEG multi-level semantics

Decoding visual information from electroencephalography (EEG) signals is crucial in neuroscience and artificial intelligence. While existing methods have been able to extract high-level features such as object categories, the capability of extracting fine-grained attributes, such as color distribution, remains insufficient. In this work, we propose EEG2IM, a novel framework that integrates multi-level EEG semantic features to guide a diffusion model for fine-grained image generation.

In EEG2IM, high-level semantic features are extracted by using a high-level semantic encoder, trained via knowledge distillation, and learnt from the ResNet50 network by response-based and feature-based methods, respectively; and low-level features, i.e., fine-grained attributes, are extracted by using a low-level semantic encoder and aligned with image features from an autoencoder via joint training. These multi-level features are incorporated into a diffusion model using Feature-wise Linear Modulation (FiLM), which enables precise control over image synthesis while preserving both semantic consistency and fine-grained details.

EEG2IM was validated on ImageNet-40 and ImageNet-4, demonstrating superior performance in classification and image generation. It achieved 99.95% accuracy on ImageNet-40 and 92.55% accuracy on ImageNet-4. For image generation, EEG2IM outperformed existing methods, achieving an Inception Score (IS) of 17.58 and Fréchet Inception Distance (FID) of 52.84 on ImageNet-40, and an IS of 8.79 with an FID of 19.49 on ImageNet-4. These results highlight EEG2IM's ability to capture both high-level semantics and low-level details, advancing fine-grained EEG-based image generation.