xpixelgroup / classsr Goto Github PK
View Code? Open in Web Editor NEW(CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic
(CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic
hello, where is the training data placed
如题
First of all, thank you for your excellent work!
Secondly I have some questions that I don't quite understand and would like to ask.
Is your proposed data feature accelerated super-resolution network applicable to any super-resolution network, because I want the framework to reduce the training computation of other networks, how can I do it?
If I use the framework to reduce the training computation of other networks do I need to train a pre-trained model first and how to train it?
I saw that the network did some data preprocessing work before training, can you explain the role of these work, why do I need to downsample the images to 0.6, 0.7, 0.8, 0.9?
I see that the classification model for simple, medium and complex classification is chosen by the criterion of psnr, what images are selected to calculate the psnr?
When testing, I see you say that some datasets are too small for crop testing, how many times the minimum dataset can be tested?
I look forward to your reply, thank you!
请问一下是通过将DIV8K进行×4 downsampling操作得到Test 2K吗
hello, I tested 100 images from DIV2k(index from 800--900), use ClassSR-FSRCNN pretrained model, it seems that the output HR images(1232x1048, 0801_x4.png)'s revolution differs from the GT(2040x1356, 0801.png) . so is there something with me or the code ?
Recently, I am impressed by your brilliant and impressive idea and work in this paper:" ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic". It groundbreakingly accelerates SR models by classing sub-images, which attracts me a lot.
Therefore, I download the released code on Github, follow the guidance in README.md, and test it. However, the results are weird. Both PSNR and SSIM are acceptable in branches testing, nevertheless, these indexes drop to ~9dB when it comes to the whole model, which confused me a lot.
And when I try to train the model, it shows that there is a lack of LMDB file to handle validating set "Set5" during the training procedure. 2 affixed pictures demonstrate the situation. I also try to read through the above paper several times, but little things are related to this problem.
So in a word, there are 2 problems I need your valuable help with:
Is there something wrong when I test the model?
How can I tackle LMDB issues?
Thank you for your reading and priceless help!
{"mode":"full","isActive":false}
Is this normal?
作者你好,我注意到你的损失并没有使用交叉熵等分类常用的损失,而是平均整体概率以及放大某一概率。
那么文章开始用网络对数据划分的作用体现在哪里呢?
训练中是否强化了分类网络将某一未知类别的数据归化到它已知的标签类别呢?
您好!我有两个问题需要请教:
def cal_FLOPs(which_model,num_ress):
if which_model =='classSR_3class_fsrcnn_net':
flops = ((141 * num_ress[0] + 304 * num_ress[1] + 468 * num_ress[2]) / sum(num_ress))+7.4
percent=flops/468
elif which_model=='classSR_3class_carn':
flops = (0.38 * num_ress[0] + 0.77 * num_ress[1] + 1.15 * num_ress[2]) / sum(num_ress)
percent = flops / 1.15
elif which_model=='classSR_3class_srresnet':
flops = (1.66 * num_ress[0] + 3.44 * num_ress[1] + 5.20 * num_ress[2]) / sum(num_ress)
percent = flops / 5.20
elif which_model=='classSR_3class_rcan':
flops = (10.33 * num_ress[0] + 19.90 * num_ress[1] + 32.60 * num_ress[2]) / sum(num_ress)
percent = flops / 32.60
return flops,percent
https://github.com/Xiangtaokong/ClassSR/blob/0ebefe75b2d5d08cffd8613c78d6dbb323dbd2c5/codes/models/archs/arch_util.py#L103-L114
I face train problems when train class_carn, finally I located the problem in above code.
In CARN official repo, I found there are no bn layer, So may be the CARN model is incorrectly implemented?
https://github.com/nmhkahn/CARN-pytorch/blob/master/carn/model/ops.py#L33-L48
你好,请问一下:我想训练一个新的训练集,那么划分训练集所需的divide_train.log要如何生成?
您好,DIV2K下载页面有多个数据集,请问你用的是哪个呢?
Hello, I have now generated sub-images. I want to use the generated sub-images to train my models of different branches. How can I create a divide_train.log file suitable for my model?
Thanks for your great work.
line 45 and line 47, might be a valid check for height and width?
你好,请问一下为什么我将三个分支的预训练模型修改为自己的模型,然后没有切分子图,在训练分类器的时候class_loss=0, average_loss=1?
data_augmentation.py has some problem.
such as (_get_paths_from_images) (imresize)
大佬您好,请问如何可视化演示图像呢?
请问验证集set5需要做什么处理吗
下载的test2k数据集应该放在什么目录下面
Hi, first of all thanks for your work!
I have question about custom data. If I have HR images with resolution 512384 and lowres with 256192, and I want to convert LR->HR, what parameters should I change change in test_ClassSR_RCAN.yml for example apart from scale ( if I change only scale I've got this error:
sr_img[i*step*self.scale:i*step*self.scale+patch_size*self.scale,j*step*self.scale:j*step*self.scale+patch_size*self.scale,:]+=sr_list[index]
ValueError: operands could not be broadcast together with shapes (64,64,3) (128,128,3) (64,64,3)
Thanks for your awesome work. When i run the trian.py with multi card, I find the tb_logger have a bug with rank > 0.
Here, rank is checked.
train.py#L125
if rank <= 0:
logger.info('Number of val images in [{:s}]: {:d}'.format(
dataset_opt['name'], len(val_set)))
But not check in here.
train.py#L215
# tensorboard logger
if opt['use_tb_logger'] and 'debug' not in opt['name']:
tb_logger.add_scalar('psnr', avg_psnr, current_step)
It should change to
# tensorboard logger
if rank <= 0 and opt['use_tb_logger'] and 'debug' not in opt['name']:
tb_logger.add_scalar('psnr', avg_psnr, current_step)
the train_ClassSR has same bug.
您好,Class-Module我已经训练了两万多个Iteration了,但是验证的时候所有的图片还是都只进入一个分支,并没有平均进入三个分支,请问这样可能是什么原因造成的呢,(initial learning rate设置为10-3,the weights w1, w2, w3 are set to 2000,1, 6)
您好,我对您的工作非常感兴趣。
我注意到您的模型在训练分类器的时候设置的loss权重 l1 : class : avg 是 2000:1:6 。我想请问这部分是怎么设置的。其次,我想请教一下在训练过程中,添加权重后的各个损失的比例要接近1:1:1还是在l1的比重更大呢?
期待您的回复,感谢!
作者您好,我用您的模型跑了一些测试代码,但生成的图都是裁剪过的,可以取消裁剪GT图吗?(我改了crop_border为none也还是会裁剪掉)
Traceback (most recent call last):
File "train_ClassSR.py", line 347, in
main()
File "train_ClassSR.py", line 277, in main
model.test()
File "/home/dsm/ClassSR-main/codes/models/ClassSR_model.py", line 199, in test
self.num_res = self.print_res(type_res)
File "/home/dsm/ClassSR-main/codes/models/ClassSR_model.py", line 352, in print_res
for i in torch.max(type_res, 1)[1].data.squeeze():
File "/home/dsm/anaconda3/envs/nsw2/lib/python3.7/site-packages/torch/tensor.py", line 451, in iter
raise TypeError('iteration over a 0-d tensor')
TypeError: iteration over a 0-d tensor
While running the training code in classSR it occurs an error on this file train_ClassSR.py in line277.How could i solve this .Thanks in advance
Thank you for your amazing work!
I've got a little question about the way you combine the output sub-images. Due to the ill-posed nature, simply concat sub-images may cause unnatural effects. I noticed in your code that you have done the following transformation sr_img[:,j*step*self.scale:j*step*self.scale+(patch_size-step)*self.scale,:]/=2
. What considerations led you to carry out this operation and is there any basis for it?
作者您好,我看了论文中提到先对原始数据下采样 0.9, 0.8, 0.7, 0.6,作为HR,再对这些数据下采样4倍得到LR,请问这样做是出于什么原因呢?
Thank for the great work.
I have trained on a custom dataset by using ClassSR repo. But not able to find the trained weights after each epochs . Should i wait to complete the entire epochs to get the trained weights ?
作者你好,首先感谢带来非常优秀的工作!
该项工作在理论计算量(flops)上有着非常明显的加速,但是在测试中发现,切块,判断,多分支推理的pipline实际消耗的推理时间似乎并没有优势,甚至更耗时。
不知道作者是否有相关的优化经验可以交流下,万分感谢!
我想使用别的数据集进行训练,想问一下该文件里面的PSNR、SSIM、PSNR_Y、SSIM_Y怎么获取?
Really interested in your work, really looking forward to see the paper
Hello, I have downloaded the DIV8K, I want to know how to generate the datasets (includes test2k(LR/HR), test4k(LR/HR), and test8k(LR/HR))?
NICE WORK!
I train classSR structure to process the denoising task. the branchs adopt small network, which calculation cost is only about 5 to 16 GMAC.
There are two problems that bother me:
First, the training speed is too slow. I use two 3090 cards to train models and it takes 12 minutes to run 100iter. However, when I increased to four 3090 cards, the speed did not increase, and the utilization rate of each GPU was very low. Could you please give me some suggestions?
Second, because the training is too slow, I used .pth at 30000iter for testing, and the classification is uneven. The image results have a strong sense of demarcation, which is mainly because different patches take different branches.Will this phenomenon be improved in the next training?
请问可以使用这个网络进行除了x4以外其他倍率的SR操作吗?如果可以,大致需要做怎样的修改呢;如果不行,需要做什么别的工作呢?
E:\Anaconda\envs\torch11\python.exe D:/迅雷下载/ClassSR-main/codes/data_scripts/extract_subimages_test.py
process GT...
mkdir [D:/迅雷下载/ClassSR-main/data0/xtkong/data/DIV2K_valid_HR_sub/GT] ...
[ ] 0/100, elapsed: 0s, ETA:
Start...
All subprocesses done.
process LR...
mkdir [D:/迅雷下载/ClassSR-main/data0/xtkong/data/DIV2K_valid_HR_sub/LR] ...
[ ] 0/100, elapsed: 0s, ETA:
Start...
All subprocesses done.
Traceback (most recent call last):
File "D:/迅雷下载/ClassSR-main/codes/data_scripts/extract_subimages_test.py", line 143, in
main()
File "D:/迅雷下载/ClassSR-main/codes/data_scripts/extract_subimages_test.py", line 66, in main
assert len(data_util._get_paths_from_images(save_GT_folder)) == len(
File "D:\迅雷下载\ClassSR-main\codes\data\util.py", line 31, in _get_paths_from_images
assert images, '{:s} has no valid image file'.format(path)
AssertionError: D:/迅雷下载/ClassSR-main/data0/xtkong/data/DIV2K_valid_HR_sub/GT has no valid image file
Thanks, it is a interesting work. I have some questions to ask. Looking forward to your answer.
(1) You propose the Average-Loss and Class-Loss to train class module, It seems the class module will be trained to output the class result simple:medium:hard with 1:1:1. But in your paper, it is not.
(2) After trained the Class Module independently, why train Class & SR Module jointly? I think maybe the pre-trained SR Module has been the best?
(3) how many 2080Ti GPUS need to train(and total memory)? About how long it takes to train?
请问一下,论文里“Then we densely crop 1.59M sub-images with size 32 × 32 from LR images. These sub-images are
equally divided into three classes (0.53M for each) according to their PSNR values through MSRResNet. ”
这里的1.59M户和0.53M 指的是文件大小还是其他?
20-04-14 00:47:54.197 - INFO: 0558_s290 - PSNR: 26.473503 dB; SSIM: 0.751480; PSNR_Y: 27.808832 dB; SSIM_Y: 0.769544.
20-04-14 00:47:54.248 - INFO: 0558_s291 - PSNR: 26.580721 dB; SSIM: 0.764139; PSNR_Y: 27.929448 dB; SSIM_Y: 0.785679.
20-04-14 00:47:54.300 - INFO: 0558_s292 - PSNR: 25.955706 dB; SSIM: 0.801685; PSNR_Y: 27.237049 dB; SSIM_Y: 0.822988.
divide_train.log 里面scale=1的图名字直接是0558_s290,0558_s291。。。。但是extract_subimages_train.py生成的文件夹DIV2K_scale_sub里面的图名字是1_0558_s290,所以在运行divide_sub_images_train.py时会报错
论文里说“To prepare the training data, we first downsample3 the original images with scaling factors 0.6, 0.7, 0.8, 0.9 to generate the HR images.” 这个意思时scale=1的图不参与训练吗?
请问一下如何复现论文中的baseline结果
FSRCNN-O论文里提到baseline是25.61dB。然后我试了两种方式来复现:( 1)用你的训练集训自己FSRCNN然后再跑test2k(与classsr的区别是直接对整张图超分,没有分解和组合图片的操作);(2)用你提供的预训练的classsr模型,然后测试时候强制都分到最复杂的超分分支上。
这两种结果获得的PSNR都比论文里提到的FSRCNN-O要高,方式(1)高0.4,方式(2)高0.2。
(我另外试了其他几个backbone,复现Original module的时候都有挺大差异的,因此想了解一下论文中的baseline是如何获得的 :)
您好,请问模型参数量Parameters以及FLOPs是用什么方法测的啊
我用profile函数测得好像跟论文不一样
大佬您好!请问生成图像的时候,怎么把gpu利用起来呀,难道是我只有一块gpu没有设置好吗?
大佬您好,我在训练RCAN的一个branch的时候经常会在八倍上出现tensor上的问题,但是其他的都能正常的跑,改了很久都没有改出来,不知道您是否方便分享一下八倍的RCAN的三个分支的预训练模型,很抱歉麻烦您!
作者您好,这是一篇非常不错的文章。我有一个问题想问一下,训练的第三阶段(联合训练阶段)的学习率和前两阶段保持一致吗,都是1e-3吗?
Hi, @Xiangtaokong ,
Thanks very much for your interesting work.
Currently I want to reproduce your work. I follow the readme and finish the data processing. while during run training command "python train_ClassSR.py -opt options/train/train_ClassSR_FSRCNN.yml", it seems to be stuck. Can you give me some suggestions?
作者您好,我想用人脸的图像来试试您的这个方法的效果。所以我想知道如何如何用我自己的人脸数据集生成divide_train.log,谢谢啦~
The following problems occurred when testing with Test8K
The output is all tensors, except for the last one, numpy, which feels like a problem with 'rlt', how can we solve it, please?
error:
Testing [test8k]...
<models.ClassSR_model.ClassSR_Model object at 0x7fc340af9470>
Traceback (most recent call last):
File "/root/PycharmProjects/ClassSR-main/codes/test.py", line 70, in
sr_img = util.tensor2img(visuals['rlt']) # uint8 question
File "/root/PycharmProjects/ClassSR-main/codes/utils/util.py", line 137, in tensor2img
*tensor = tensor.squeeze().float().cpu().clamp_(min_max) # clamp
AttributeError: 'numpy.ndarray' object has no attribute 'float'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.