inference with jpg files

[xinntao]

inference with jpg files

[xinntao]

@suke27 @rahidz
Yes, if the input images are JPEG images, which means it was compressed by JPEG algorithms, the output is "blocky" with checkboard.

During training, the ESRGAN has not seen these blocky JPEG artifacts and was not trained with the objective to remove them. Therefore, in testing, the ESRGAN "enhancing" the jpg artifacts already present, as @rahidz said.

Turning it to PNG format cannot remove these checkerboard (8 pixels by 8 pixels) in JPEG compression. After upsampling the image, the checkerboard artifacts appear (8*4=32pixels, so you will see 32 by 32 pixels checkerboard artifacts).

It is a limitation that ESRGAN (also other SR algorithms using the bicubic downsampling kernels) cannot handle the problems. Acturally, the ESRGAN (and many other SR algorithms) is limited to a very strong assumption - the perfect bicubic downsampling kernels. Howevere, in real world, the input images have diverse down-sampling kernels and are also with blur, JPEG compression and noise.

So the algorithms are not satisfactory for images in real world. These problems are also the research topics that the community wants to solve.

You can try to improve it by 1) removing the JPEG artifacts first by other algorithms like ARCNN; 2) finetuning the network with datasets with JPEG artifacts, so that the ESRGAN can handle them.

[rahidz]

For some jpg files I try, the output is very “blocky”’ like there’s small but very noticeable square blocks covering the entire image. I assume this is simply ESRGAN “enhancing” the jpg artifacts already present, since this doesn't noticeably occur for all jpg files in my experience.

I’ve noticed slight improvement by converting the jpg to a png before I start, but it’s still not great.

[suke27]

@xinntao for suggestion 2) finetuning the network with datasets with JPEG artifacts, so that the ESRGAN can handle them.
I try to finetune using jpg dataset for SRGAN, but not work well, also block and noise exist
but same training method for EDSR works well.

[xinntao]

@suke27 You can try to first pre-train the network with L2/L1 loss with JPEG compressed images. And then use the pre-trained model to train a GAN-based model.

I think, L2/L1 loss is a clear and effective loss to guide the network to remove the JPEG compression artifacts (Therefore, finetuning with jpg dataset works well for EDSR). But for SRGAN, the VGG + Adv loss may be not enough to remove the JPEG artifacts. So I think first pre-training the network with JPEG compressed images to let the network learn to remove JPEG artifacts will work.

[neilthefrobot]

I was able to get around this by taking my HR training set, down sampling 4x, then converting it to .jpg with a low quality (high compression) and using that as my LR set. This way the network is seeing jpg artifacts as inputs and a non jpg artifact version as a target and it learns to convert between them. It actually worked very well too.