Extracting Semantic Knowledge From GANs With Unsupervised Learning.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Unsupervised learning is advancing rapidly, with growing interest in representations from generative models like Generative Adversarial Networks (GANs).
• Prior research indicates that GANs encode semantic information linearly separable within their feature maps.
• This linear separability provides a foundation for effective clustering of GAN-generated features.

Purpose of the Study:

• To propose a novel clustering algorithm, KLiSH, designed to leverage the linear separability of GAN features.
• To demonstrate KLiSH's capability in extracting fine-grained semantic information from GANs trained on diverse object datasets.
• To utilize KLiSH for synthesizing paired image-segmentation datasets for downstream applications.

Main Methods:

• Developed KLiSH, a clustering algorithm exploiting the linear separability of GAN features.
• Applied KLiSH to GANs trained on datasets including cars, portraits, and animals.
• Synthesized paired image-segmentation datasets using KLiSH-generated features and GAN sampling.
• Trained semantic segmentation and image-to-image translation networks on the synthesized datasets.

Main Results:

• KLiSH successfully clustered GAN features, revealing fine-grained semantics across various object categories.
• Generated paired image-segmentation datasets enabled unsupervised semantic segmentation on real images.
• Trained image-to-image translation models achieved semantic-conditional synthesis without human annotations.

Conclusions:

• The KLiSH algorithm effectively extracts semantic information from GANs through feature clustering.
• Synthesized datasets from KLiSH facilitate unsupervised semantic segmentation and semantic-conditional image synthesis.
• This approach offers a novel pathway for leveraging GANs in downstream computer vision tasks without manual annotation.