luissen/esrt

esrt's Issues

where is the 'architecture' in folder model ?

In train.py, line 6: from model import architecture, esrt.
But there is no 'architecture' in the model folder ?

Pretrained models

Hi! Do we have pretrained models for this work? It would be nice to run inference on a couple of images to see what would we obtain, plus, a good starting point for training. Thanks!

Try adding MLABlock to my model

when training this model ,
x = x + self.atten(self.norm1(x))
the expected data shape is [*,64], i modified the data according to the expected size .
"def forward(self, x):
x = self.reduce(x)
print(x.shape)
B, N, C = x.shape"
But the expected data shape at here is three-dimensional

error of psnr/ssim

sorry for bothering,

when i run the original code, the size of 'im_pre' and 'im_label' is different, and always 'im_pre' is 6-8 pixels less than the 'im_label'. May i ask why this happened and how to solve this issue?

thanks in advance for your support and attention.

DataLoader worker (pid(s) 9504, 11496, 19564, 12924, 14856, 1256, 20244, 7752) exited unexpectedly

RuntimeError Traceback (most recent call last)
e:\esrt\train.py in
244 epoch_start = datetime.datetime.now()
245 valid(args.scale)
--> 246 train(epoch)
247 if epoch%10==0:
248 save_checkpoint(epoch)

e:\esrt\train.py in train(epoch)
131 utils.adjust_learning_rate(optimizer, epoch, args.step_size, args.lr, args.gamma)
132 print('epoch =', epoch, 'lr = ', optimizer.param_groups[0]['lr'])
--> 133 for iteration, (lr_tensor, hr_tensor) in enumerate(training_data_loader, 1):
134
135 if args.cuda:

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in next(self)
519 if self._sampler_iter is None:
520 self._reset()
--> 521 data = self._next_data()
522 self._num_yielded += 1
523 if self._dataset_kind == _DatasetKind.Iterable and \

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
1184
1185 assert not self._shutdown and self._tasks_outstanding > 0
-> 1186 idx, data = self._get_data()
1187 self._tasks_outstanding -= 1
1188 if self._dataset_kind == _DatasetKind.Iterable:

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _get_data(self)
1150 else:
1151 while True:
-> 1152 success, data = self._try_get_data()
1153 if success:
1154 return data

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout)
1001 if len(failed_workers) > 0:
1002 pids_str = ', '.join(str(w.pid) for w in failed_workers)
-> 1003 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
1004 if isinstance(e, queue.Empty):
1005 return (False, None)

RuntimeError: DataLoader worker (pid(s) 9504, 11496, 19564, 12924, 14856, 1256, 20244, 7752) exited unexpectedly

kidly help to resolve this error. I am also not able to use GPU with the code

Model weights used in paper

Hi! Do we have the trained model weights for this work? It would be nice to run inference on a couple of images with the same performance as in the paper.

file image.py not found

While executing the code on Spyder notebook i got the following error:
run /Image Transformer/esrt/train.py --scale 2 --patch_size 96
Traceback (most recent call last):

File C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\magics\execution.py:696 in run
filename = file_finder(fpath)

File C:\ProgramData\Anaconda3\lib\site-packages\IPython\utils\path.py:91 in get_py_filename
raise IOError('File %r not found.' % name)

OSError: File '/Image.py' not found.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

Input In [4] in <cell line: 1>
get_ipython().run_line_magic('run', '/Image Transformer/esrt/train.py --scale 2 --patch_size 96')

File C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py:2294 in run_line_magic
result = fn(*args, **kwargs)

File C:\ProgramData\Anaconda3\lib\site-packages\IPython\core\magics\execution.py:707 in run
raise Exception(msg) from e

Exception: File '/Image.py' not found.

Cannot reproduce the result in the paper.

Hi, I use python train.py --scale 4 --patch_size 192 to train the ESRT but get 32.07/0.8919 (PNSR/SSIM) on Set5, which has a certain difference with the results in the original paper. I check the hyper-parameters in training code, which is same with them in paper. I don't know what went wrong, can you please tell me the training parameters when training the model ？

About the calculate Flops and GPU memory cost

Thanks foy your excellent works. I have trained on your relased codes, and wants to calculate Flops about your model. As you report in paper, I used 1280x720 to calculate flops, but have a bug about out of memory. And I find that your models seems cost a expensive GPU memory when the image sizes grows. I only can inference 256x256 images X4-SR on a single Titan RTX GPU with 24G memory. Counterintuitively, the models only a single efficient Transformer blocks but cost expensive memory. why this model can not inference on lager images?

what is the difference between Feature Split (FS) in EMHA and window-attention?

Thank you for your work.

after reading your paper, I have a question.

In Feature Split (FS) of sec. 3.2.2 Efficient Transformer, I was confused with the difference between this FS and window-attention (from Swin-Transformer).

Your FS splits the features into N/s x N/s, and window-attention (of Swin-Transformer) splits the features into N/M x N/M, where M is window size.

self-attention is calculated within N/s x N/s (by FS) and N/M x N/M (by window-partitioning), respectively.

s (in FS) and M (in window-attention) can be different in that the values differ, but I don't understand the mechanism differences between them.

Once more, thank you for your hard work.

Hi

请问你们的文章发表在什么期刊或者论文上，谢谢

High-frequency Filtering Module (HFM)相关代码

您好，我对您文章中提到的High-frequency Filtering Module (HFM)很感兴趣，但是在公开的代码中没有找到相关的部分，请问哪里是实现该功能的代码呢，谢谢

not find module "utils"

hi! I want to run your code on my computer, but it tells me, not have "utils" this module. so what should I do?

ESRT in remote sensing SR

Thanks for your wonderful work first.
When I use this model in remote sensing SR(AID dataset)，I find the the value psnr is around 19 from the epoch1 to the end of train in scale x2,x3,x4.
I would be appreciated if you could give me some help.

none

none1

"scale factor was wrong in my original version" problem

Hi, what is mean "NOTE scale factor was wrong in my original version, can set manually to be compat with prev weights" in class EffAttention? can you explain more? and how set manually?

What is common.Scale(1) means?

class Scale(nn.Module):
    def __init__(self, init_value=1e-3):
        super().__init__()
        self.scale = nn.Parameter(torch.FloatTensor([init_value]))

    def forward(self, input):
        return input * self.scale

When the self.scale=1, does this option does nothing?
Why do we need this layer?

Real SR ---training

how to train real image? Here giving the traing of div2k, but no RealSR