際際滷

際際滷Share a Scribd company logo

Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy

JaeJun Yoo
JaeJun Yoo

Paper:https://arxiv.org/abs/2004.00448 Github:https://github.com/clovaai/cutblur Youtube: https://youtu.be/MQR5mnsCvbM

1 of 17
Downloaded 11 times
Rethinking Data Augmentation for Image Super-resolution : A Comprehensive Analysis and a New Strategy; CutBlur & MoA Jaejun Yoo*, Namhyuk Ahn*, and Kyung-Ah Sohn
Motivation * Various levels of vision tasks High-level: Semantic recognition (e.g., classification, object detection) Mid-level: Super-pixel (e.g., segmentation) Low-level: Pixels, Edges, Colors (e.g., super-resolution, denoising) ????? MixUp, Cutout, CutMix
Analysis on existing DA methods Sharp transitions, mixed image contents or losing the relationships of pixels can degrade SR performance. e.g., Cutout fails (discarding pixels) and every feature method fails (manipulation). Training curves when applied feature DAs
Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully.
Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. Cutout: Original setting (drop 25% of of pixels in a rectangular shape) significantly degrades the performance because it erases spatial information too much. However, erasing tiny amount of pixels (0.1% random pixels) boosts the performance (2~3 pixels of 48x48 input patch) Cutout
Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. Mixup & CutMix: Improvements of using CutMix are marginal. We suspect this happens because CutMix generates a drastic sharp transition between two different images. Improvements of using Mixup is better than CutMix but it still generates unrealistic image and affects to the image structure. Mixup CutMix
Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. CutMixup: To verify our hypothesis, we combine benefits of Mixup and CutMix; CutMixup. CutMixup provides various boundary cases while minimizes the sharp transition by retaining partial cues as Mixup does. CutMixup
Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. Blend & RGB permutation: To push further, we tried a constant blending and RGB channel permutation, which turn out to be very simple but effective strategies showing big performance enhancement (dB). Note that both methods do not incur any structure modification to an image. BlendRGB perm.
CutBlur
CutBlur What does the model learn from CutBlur? CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained without CutBlur and its error residual () Error residual ()Output
CutBlur What does the model learn from CutBlur? CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained CutBlur and its error residual () Error residual ()Output with
Improved generalization: over-sharpening Super-resolution (SR) Trained on 4 scale factor dataset and tested on different scale factor (2)
Improved generalization: over-smoothing Denoising Trained on severe noise (! = 70) & tested on mild noise (! = 30).
Improved generalization: over-removal JPEG artifact removal Trained on a mild compression rate & tested on a severe compression rate
Mixture of Augmentation (MoA) During the training phase Randomly select single augmentation at every step. (among the curated DA list) Apply it!
Comparison on diverse benchmark models and datasets SRCNN (0.07M) ECCV14, CARN (1.14M) ECCV18, RCAN (15.6M) ECCV18, EDSR (43.1M) CVPRW17 DIV2K (synthetic), RealSR (real-world) Our method shows consistent improvement for different models (parameters) and datasets (different environments and size):
Code: https://github.com/clovaai/cutblur Paper: https://arxiv.org/abs/2004.00448 QR code for code & paper For more details, please visit our website:

More Related Content

Rethinking Data Augmentation for Image Super-resolution: A Comprehensive Analysis and a New Strategy

  • 1. Rethinking Data Augmentation for Image Super-resolution : A Comprehensive Analysis and a New Strategy; CutBlur & MoA Jaejun Yoo*, Namhyuk Ahn*, and Kyung-Ah Sohn
  • 2. Motivation * Various levels of vision tasks High-level: Semantic recognition (e.g., classification, object detection) Mid-level: Super-pixel (e.g., segmentation) Low-level: Pixels, Edges, Colors (e.g., super-resolution, denoising) ????? MixUp, Cutout, CutMix
  • 3. Analysis on existing DA methods Sharp transitions, mixed image contents or losing the relationships of pixels can degrade SR performance. e.g., Cutout fails (discarding pixels) and every feature method fails (manipulation). Training curves when applied feature DAs
  • 4. Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully.
  • 5. Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. Cutout: Original setting (drop 25% of of pixels in a rectangular shape) significantly degrades the performance because it erases spatial information too much. However, erasing tiny amount of pixels (0.1% random pixels) boosts the performance (2~3 pixels of 48x48 input patch) Cutout
  • 6. Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. Mixup & CutMix: Improvements of using CutMix are marginal. We suspect this happens because CutMix generates a drastic sharp transition between two different images. Improvements of using Mixup is better than CutMix but it still generates unrealistic image and affects to the image structure. Mixup CutMix
  • 7. Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. CutMixup: To verify our hypothesis, we combine benefits of Mixup and CutMix; CutMixup. CutMixup provides various boundary cases while minimizes the sharp transition by retaining partial cues as Mixup does. CutMixup
  • 8. Analysis on existing DA methods DA methods in pixel space bring some improvements when applied very carefully. Blend & RGB permutation: To push further, we tried a constant blending and RGB channel permutation, which turn out to be very simple but effective strategies showing big performance enhancement (dB). Note that both methods do not incur any structure modification to an image. BlendRGB perm.
  • 10. CutBlur What does the model learn from CutBlur? CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained without CutBlur and its error residual () Error residual ()Output
  • 11. CutBlur What does the model learn from CutBlur? CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the necessary region. Super-resolution results of a model (EDSR) trained CutBlur and its error residual () Error residual ()Output with
  • 12. Improved generalization: over-sharpening Super-resolution (SR) Trained on 4 scale factor dataset and tested on different scale factor (2)
  • 13. Improved generalization: over-smoothing Denoising Trained on severe noise (! = 70) & tested on mild noise (! = 30).
  • 14. Improved generalization: over-removal JPEG artifact removal Trained on a mild compression rate & tested on a severe compression rate
  • 15. Mixture of Augmentation (MoA) During the training phase Randomly select single augmentation at every step. (among the curated DA list) Apply it!
  • 16. Comparison on diverse benchmark models and datasets SRCNN (0.07M) ECCV14, CARN (1.14M) ECCV18, RCAN (15.6M) ECCV18, EDSR (43.1M) CVPRW17 DIV2K (synthetic), RealSR (real-world) Our method shows consistent improvement for different models (parameters) and datasets (different environments and size):
  • 17. Code: https://github.com/clovaai/cutblur Paper: https://arxiv.org/abs/2004.00448 QR code for code & paper For more details, please visit our website:
AboutCopyright
息 2025 際際滷