SaGAN:Generative Adversarial Net

作者: yestinl | 来源:发表于2018-11-06 19:20 被阅读0次

SaGAN:Generative Adversarial Net
Duplex Generative Adversarial Ne
GAN综述|Generative Adversarial Net
Generative Adversarial Nets翻译[上]
深度学习笔记（十二）—— GAN-1
Conditional GAN python 实现
论文阅读之 Generative Adversarial Net
GAN对抗网络入门教程
Generative Adversarial Network i
医学图像中GAN2019综述

Problem

Traditional GAN methods directly operate on the whole image, and inevitably change the attribute-irrelevant regions
The performance of traditional regression methods heavily depends on the paired training data, which are however quite difficult to acquire

Relative work

ResGAN: learning the residual image avoids changing the attribute-irrelevant region by restraining most regions of the residual image as zero.
Improvement:This work is quite insightful to enforce the manipulation mainly concentrate on local areas especially for those local attributes.
Drawback: the location and the appearance of target attributes are modeled in single sparse residual image which is actually hard for a favorable optimization than modeling them separately

Method

SaGAN: only alter the attribute specific region and keep the rest unchanged
The generator contains an attribute manipulation network (AMN) to edit the face image, and a spatial attention network (SAN) to localize the attribute-speciic region which restricts the alternation of AMN within this region.

Contribution

The spatial attention is introduced to the GAN framework, forming an end-to-end generative model for face attribute editing (referred to as SaGAN),which can only alter those attribute-speciic region and keep the rest irrelevant region remain the same.
The proposed SaGAN adopts single generator with attribute as conditional signal rather than two dual ones for two inverse face attribute editing.
The proposed SaGAN achieves quite promising results especially for those local attributes with the attribute-irrelevant details well preserved. Besides, our approach also benefits the face recognition by data augmentation.

Generative Adversarial Network with Spatial Attention

SaGAN

notation	meaning
$I$	input image
$\hat{I}$	output image
$I_a$	an edited face image output by AMN
$c$	attribute value
$c_g$	ground truth attribute label of the real image $I$
$D_{src}(I)$	probability of an image $I$ to be a real one
$D_{cls}(c\|I)$	probability of an image $I$ with the attribute $c$
$F_m$	an attribute manipulation network (AMN)
$F_a$	a spatial attention network(SAN)
$b$	a spatial attention mask, used to restrict the alternation of AMN within this region
$\lambda_1$	balance parameters
$\lambda_2$	balance parameters
$λ_{gp}$	hyper-parameters control the gradient penalty, default = 10

the goal of face attribute editing is to translate I into an
new image $\hat{I}$ , which should be realistic, with attribute c and look the same as the input image excluding the attribute-specific region

Discriminator

Two objectives, one to distinguish the generated images from the real ones, and another to classify the attributes of the generated and real images
The two classifiers are both designed as a CNN with softmax function, denoted as $D_{src}$ and $D_{cls}$ respectively.
The two networks can share the first few convolutional layers followed by distinct fully-connected layers for different classifications
$\mathcal{L}_{src}^D = \mathbb{E}_I[logD_{src}(I)]+\mathbb{E}_\hat{I}(log(1-D_{src}(\hat{I}))]\tag{1}$
$\mathcal{L}_{cls}^D = \mathbb{E}_{I,c^g}[-logD_{cls}(c^g|I)]$
discriminator D:
$\min \limits_{D_{src},D_{cls}} \mathcal{L}_D = \mathcal{L}_{src}^D+\mathcal{L}_{cls}^D$

Generator

G contains two modules, an attribute manipulation network(AMN) and a spatial attention network(SAN)
AMN focuses on how to manipulate and SAN focuses on where to manipulate.
The attribute manipulation network takes a face image $I$ and an attribute value $c$ as input, and outputs an edited face image $I_a$
$I_a = F_m(I,c)$
The spatial attention network takes the face image $I$ as input, and predict a spatial attention mask $b$ , which is used to restrict the alternation of AMN within this region
Ideally, the attribute-specific region of $b$ should be 1, and the rest regions should be 0.
Regions with non-zeros attention values are all regarded as attribute-specific region, and the rest with zero attention values are regarded as attribute-irrelevant region
$b = F_a(I)$
the attribute-specfiic regions are manipulated towards the target attribute while the rest regions remain the same
$\hat{I} = G(I,c) = I_a \cdot b + I \cdot (1-b)$

To make the edited face image $\hat{I}$ photo-realistic: an adversarial loss is designed to confuse the real/fake classifier
$\mathcal{L}^G_{src} = \mathbb{E}_{\hat{I}}[[-logD_{src}(\hat{I})]]\tag{2}$
To make $\hat{I}$ be correctly with target attribute $c$ : an attribute classification loss is designed to enforce the attribute prediction of $\hat{I}$ from the attribute classifier approximates the target value $c$
$\mathcal{L}_{cls}^G = \mathbb{E}_\hat{I}[-logD_{cls}(c|\hat{I})]$
To keep the attribute-irrelevant region unchanged: a reconstruction loss is employed similar as CycleGAN and StarGAN
$\mathcal{L}_{rec}^G = \lambda_1\mathbb{E}_{I,c,c^g}[(||I-G(G(I,c),c^g)||_1]+\lambda_2\mathbb{E}_{I,c^g}[(||I-G(I,c^g)||_1]$
generator G
$\min \limits_{F_m,F_a} \mathcal{L}_G = \mathcal{L}_{adv}^G+\mathcal{L}_{cls}^G+\mathcal{L}_{rec}^G$

Implementation

Optimization

To optimize the adversarial real/fake classification more stably, in all experiments the objectives in Eq.(1) and Eq.(2) is optimized by using WGAN-GP
$\mathcal{L}_{src}^D = -\mathbb{E}_I[D_{src}(I)]+\mathbb{E}_\hat{I}[D_{src}(\hat{I})]+\lambda_{gp}\mathbb{E}_\tilde{I}[(||\nabla_\tilde{I}D_{src}(\tilde{I})||_2-1)^2]$

$\tilde{I}$ is sampled uniformly along a straight line between the edited images $\hat{I}$ and the real images $I$

Network Architecture

For the generator, the two networks of AMN and SAN share the same network architecture except slight difference in the input and output:

Network	Input	Output	Activation function
AMN	4-channel input, an input image and a attribute	3-channel RGB image	Tanh
SAN	3-channel input, an input image	1-channel attention mask image	Sigmoid

For the discriminator, the same architecture as PatchGAN, is used considering its promising performance.

SaGAN:Generative Adversarial Net
Problem Traditional GAN methods directly operate on the w...
Duplex Generative Adversarial Ne
About this paper Title: Duplex Generative Adversarial Net...
GAN综述|Generative Adversarial Net
2019.7 论文地址：https://arxiv.org/abs/1906.01529v1 项目地址：https...
Generative Adversarial Nets翻译[上]
Generative Adversarial Nets翻译下 code Generative Adversari...
深度学习笔记（十二）—— GAN-1
实验要求与基本流程实验要求结合理论课内容，深入理解GAN(Generative Adversarial Net...
Conditional GAN python 实现
补充 Conditional Generative Adversarial Nets in TensorFlow ...
论文阅读之 Generative Adversarial Net
论文阅读之 Generative Adversarial Nets Introduction Goodfellow...
GAN对抗网络入门教程
译：A Beginner's Guide to Generative Adversarial Networks (...
Generative Adversarial Network i
论文地址：Generative Adversarial Network in Medical Imaging: A...
医学图像中GAN2019综述
论文:Generative adversarial network in medical imaging: A r...