Image colorization
?
0 likes?414 views
CycleGAN is a framework that uses generative adversarial networks to perform image-to-image translation without paired training examples. It consists of two generators that map images from one domain to another and back, along with two discriminators that classify images as real or fake. The generators are trained to translate images such that they are classified as real by the discriminators, while also remaining consistent when translated back to the original domain. The authors demonstrate it for the task of colorizing grayscale images without paired color-grayscale image samples.
More Related Content
What's hot (20)
Similar to Image colorization (20)
![[NS][Lab_Seminar_250203]KAG-prompt (1).pptx](https://cdn.slidesharecdn.com/ss_thumbnails/nslabseminar250203kag-prompt1-250203110738-340efeca-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
Image colorization
- 1. Image Colorization Using CycleGAN Zahil Shanis Yash Saraf Sai Varun
- 2. Generative Adversarial Networks ¡ñ Generative models with two competing differentiable functions, represented by neural networks. ¡ñ Generator: Generates data from random noise using feedback from discriminator. ¡ñ Discriminator: A classifier to identify real data from fake (synthesized) data. We train the generator to create data towards what the discriminator thinks is real.
- 3. CycleGAN ¡ñ Proposed by Jun-Yan Zhu, Taesung Park, Phillip Isola and Alexei A. Efros ¡ñ Performs unpaired image to image translation. ¡ñ Unpaired translation - doesn¡¯t require a training set of aligned image pairs. ¡ñ Cycle GAN can translate an image from a source domain X to a target domain Y in the absence of paired examples.
- 4. Cycle GAN Architecture ¡ñ Architecture consists of two mappings: G : X -> Y and F : Y -> X. ¡ñ A generator G is used to translate real image from domain X to domain Y. ¡ñ A generator F is used to translate real image from domain Y to domain X. ¡ñ Discriminators (Dx and Dy) are used to discriminate real and fake images at respective domains.
- 5. Cycle GAN Cost Function ¡ñ In addition to the Generator and Discriminator losses, CycleGAN uses one more type of loss called Cycle Consistency Loss. ¡ñ This enforces that the input and generated output are recognizably the same. ¡ñ Final Objective Function is given by:
- 6. Image Colorization with Cycle GAN ¡ñ Colorize gray scale images using Cycle GAN architecture. ¡ñ Training on unpaired flowers dataset - domain X as gray scale images and domain Y as color images. Network Architecture ¡ñ Generator: A UNet like architecture with an encoder, transformer and decoder. ¡ñ Discriminator: PatchGANS which look at a ¡°patch¡± of the input image, and output the probability of the patch being ¡°real¡±. ¡ñ Trained with a batch size of 1 with Adam as the optimizer.
- 8. Network Modifications 1) Cycle GAN with Stochastic Generators ¡ñ Inter domain mapping from unpaired data need not always be one-to-one or deterministic. ¡ñ Stochastic Cycle GAN - Generates multiple color images for a single grayscale image. ¡ñ Can be achieved by modifying the generator GAB to take a vector of noise and a sample from the source domain, and generates a non-deterministic sample in the target domain. ¡ñ With different noise z ~ p(z), model can generate different domain B mappings. ¡ñ Inspired from Conditional Instance Normalization for Style Transfer paper by Huang et al. ¡ñ We are working on implementing this.
- 9. Network Modifications 2) Cycle GAN with Capsule Nets ¡ñ In CNN, Pooling layers are used to increase the field of view and predict higher order features by combining values. ¡ñ Use of Capsule Nets helps preserve hierarchical pose relationships between object parts.
- 10. Network Modifications ¡ñ Capsule Networks and GANs - Using a Capsule Network as a discriminator to better train the model to understand spatial differences. ¡ñ Papers CapsGAN, and CapsuleGAN, takes forward the idea by replacing the DCGAN discriminator with CapsuleGANs.
- 11. Conditional GAN (pix2pix) ¡ñ Performs paired image to image translation. ¡ñ In an unconditioned generative model, there is no control on modes of the data being generated. ¡ñ In the CGAN, the generator learns to generate a fake sample with a specific condition or characteristics rather than a generic sample from unknown noise distribution.
- 12. Conditional GAN (pix2pix) Training a conditional GAN Combined Loss Function
- 13. References ¡ñ Cycle GAN paper by Zhu et al - https://arxiv.org/pdf/1703.10593.pdf ¡ñ Blog Cycle GAN - https://medium.com/@jonathan_hui/gan-cyclegan-6a50e7600d7 ¡ñ Cycle GAN implementation - https://github.com/eriklindernoren/Keras-GAN ¡ñ Keras documentation - https://keras.io/ ¡ñ CapsuleGAN implementation - https://github.com/gusgad/capsule-GAN/blob/master/capsule_gan.ipynb ¡ñ CapsGAN - https://arxiv.org/abs/1806.03968 ¡ñ CapsuleGAN - https://arxiv.org/abs/1802.06167 ¡ñ Capsule Networks - https://arxiv.org/abs/1710.09829 ¡ñ Blog Capsule Networks - https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintons-capsule-networks- part-i-intuition-b4b559d1159b ¡ñ Conditional Instance Normalization - https://arxiv.org/pdf/1703.06868.pdf ¡ñ Pix2pix implemntation - https://github.com/eriklindernoren/Keras-GAN/tree/master/pix2pix ¡ñ Pix2pix - https://arxiv.org/abs/1611.07004
Editor's Notes
- #3: Conceptually speaking, discriminator in the network give guidance to generator on what data to create. As we train the networks arternatively, eventually, the discriminator identifies the tiny difference between the real and the generated, and the generator creates images that the discriminator cannot tell the difference. The GAN model eventually converges and produces data indistinguishable from real data.
- #10: Max pooling loses important relevant information about translational and rotational relationship between the objects. Using Primary Caps, the digit caps are calculated using Dynamic routing algorithms. Now, these are fed to the decoder network. The model is trained using Reconstruction loss from decoder network.
- #11: Promising results for MNIST and CIFAR datasets. Outperforms simple GAN by 2-3 percent.
- #12: In the Conditional GAN (CGAN), the generator learns to generate a fake sample with a specific condition or characteristics (such as a label associated with an image or more detailed tag) rather than a generic sample from unknown noise distribution.
- #13: Training a conditional GAN . The conditioning image, x is applied as the input to the generator and as input to the discriminator. The generator in this case is trying to learn how to colorize a black and white image. The discriminator is looking at the generator¡¯s colorization attempts and trying to learn to tell the difference between the colorizations the generator provides and the true colorized target image provided in the dataset. The discriminator, D, learns to classify between fake (synthesized by the generator) and real images. The generator, G, learns to fool the discriminator.