Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks

Abastract

This paper mainly focused on solving the problem of image-translation with unpaired images. Image-to-image translation is a class of vision and graphics problems where the goal is to learn to the mapping between an input image and an output image using a training set of aligned image pairs. However, obtaining large amount of paired images could be quite expensive. The author proposed learn a translation from source domain \(X\) to target domain \(Y\) in the absence of paired images. The goal is to learn a map \(G:X\rightarrow Y\), such that the distribution of \(G(X)\) is indistinguishable from \(Y\) using an adversarial loss. The author coupled it with reverse mapping \(F:Y\rightarrow X\) and cycle-consistency loss \(F(G(X))\approx X\)

Fig.1: Given any two unordered image collections \(X\) and \(Y\), the algorithm could automatically "translate" an image from one into the other and vice versa: (left) 1074 Monet paintings and 6753 landscape photos from Flickr; (center) 1177 zebras and 939 horses from ImageNet; right 1272 summer and 854 winter Yosemite photos from Flickr. Example application (bottom): using a collection of paintings of a famous artist, learn to render a user's photograph into there style

reference

• Image-to-image translation with conditional adversarial networks
• Deep generative image models using a laplacian pyramid of adversarial networks
• Generative visual manipulation on the natural image manifold
• Improved techniques for training gans
• Learning dense correspondence via 3dguided cycle consistency
• Unsupervised monocular depth estimation with left-right consistency

Formulation

The model trains two mapping functions \(G:\rightarrow X\) and \(F:\rightarrow X\), and associated with adversarial discriminators \(D_Y\) and \(D_X\). \(D_Y\) encourages \(G\) to translate \(X\) into outputs indistinguishable from domain \(Y\), and vice versa for \(D_X\), \(F\) and \(X\).