zhangbaijin/spa-former-shadow-removal

[IJCNN 2023 Oral]: SpA-Former:An Effective and Lightweight Transformer for Image Shadow Removal

SpA-Former:An Effective and Lightweight Transformer for Image Shadow Removal(IJCNN2023 Oral)

new

• 2023.8.4 IEEE EXPORE have published here:[https://ieeexplore.ieee.org/document/10191081}

• 2023.4.7 The paper is accepted in IJCNN 2023! Thanks for your help

• 2022.6.30 The draft is released now at http://arxiv.org/abs/2206.10910 SpA-Former:Transformer image shadow detection and removal via spatial attention.

• You can play it on Google colab Thanks for the author.

• There is SpA-Former Demo testing guidelines

Attention map of SpA-Former on ISTD dataset

Qucikly run

1. TRAIN

Modify the config.yml to set your parameters and run:

python train.py

2. TEST

First，the dataset is trained on 640x480, so you should resize test dataset to 640X480, you can use the code to resize your image bash python bigresize.py and then follow the code to test the results:

python predict.py --config <path_to_config.yml_in_the_out_dir> --test_dir <path_to_a_directory_stored_test_data> --out_dir <path_to_an_output_directory> --pretrained <path_to_a_pretrained_model> --cuda

Attention visual results is bellow:Attention visual results.

There're my pre-trained models on ISTD(./pretrained_models/ISTD/gen_model_epoch_200.pth)

3. Pretrained model

Download the pretrained model shadow-removal Google-drive and Baidu Drive 提取码：rpis

4.Test results

Our test results: Google-drive and Baidu drive 提取码：18ut

5.Evaluate

To reproduce PSNR/SSIM/RMSE scores of the paper, run MATLAB script

evaluate.m

In this section, I compares SpA-Former with several methods using peak signal to noise ratio (PSNR) and structural similarity index (SSIM) and (RMSE) as metrics on datasets ISTD.

)

ACKNOLAGEMENT

The code is updated on [https://github.com/Penn000/SpA-GAN_for_cloud_removal)]

2. DATASET

2.1. ISTD_DATASET

Click official address Build the file structure as the folder data shown. Here input is the folder where the shadow image is stored and the folder target stores the corresponding no shadow images.

./
+-- data
    +--	ISTD_DATASET
        +-- train
        |   +-- input
        |   |   +-- 0.png
        |   |   +-- ...
        |   +-- target
        |       +-- 0.png
        |       +-- ...
        +-- test
            +-- input
            |   +-- 0.png
            |   +-- ...
            +-- target
                +-- 0.png
                +-- ...

CONTACT

Contact me if you have any questions about the code and its execution.

E-mail: [email protected]

If you think this work is helpful for your research, give me a star :-D

Citations

@INPROCEEDINGS{10191081,
  author={Zhang, Xiaofeng and Zhao, Yudi and Gu, Chaochen and Lu, Changsheng and Zhu, Shanying},
  booktitle={2023 International Joint Conference on Neural Networks (IJCNN)}, 
  title={SpA-Former:An Effective and lightweight Transformer for image shadow removal}, 
  year={2023},
  volume={},
  number={},
  pages={1-8},
  doi={10.1109/IJCNN54540.2023.10191081}}

spa-former-shadow-removal's Issues

Is training dataset required to be paired (i.e. supervised learning)?

Hi, thanks for sharing the great work! I wonder if I want to finetune the model with my own dataset. My training dataset needs to be paired? i.e., each pair is a shadow image and its corresponding shadow-free image? Can the training dataset be unpaired?

Thanks!

Predict.py not working , untitled1 not found

ModuleNotFoundError: No module named 'untitled1'

关于train和val的问题

我看在eval.py中对比生成图像和目标图像的相似度时，生成图和目标图有个
img1 = np.tensordot(out.cpu().numpy()[0, :3].transpose(1, 2, 0), [0.298912, 0.586611, 0.114478], axes=1)
img2 = np.tensordot(t.cpu().numpy()[0, :3].transpose(1, 2, 0), [0.298912, 0.586611, 0.114478], axes=1)
操作，这个操作是干什么的呀？[0.298912, 0.586611, 0.114478],这个数据又是什么作用？
还有我看训练的时候还用到了Discriminator，但是我看论文中没有提到，想问一下这个的作用。
然后想问一下arXiv上论文还会不会更新呀

predict.py is not running & Cannot access to pre-trained models.

First of all thanks for your great work.

I`m interested in shadow removal and found your repository.
I have problem with run this code.

1.https://github.com/zhangbaijin/SpA-Former-shadow-removal/blob/main/pretrained_models/RICE1 Cannot access
2. Download gen_model_epoch160.pth in https://drive.google.com/drive/folders/1pxwwAfwnGKkLj-GAlkVCevbEQM4basgR and
run with this parameter error occured at gen.load_state_dict.

I think model was deprecated *.pth
Could you upload lastest pre-trained model or if i have mistake could you guide my fault??

configuration and memory?

Thanks for your great work! But I can't find any instructions for configuration, can you tell me how to set up the environment in order to run predict.py. Thanks!

gpu 编号问题

你好，使用除了0号卡训练，包括单卡和多卡好像都有问题，比如说，我规定了卡[2,3]，但是仍然显示卡是[1,2]。后来我把这个解决了。。。。

问题仍然没有解决，仍然报错。即使我将卡规定在了[2,3]，仍然说有部分数据放在了0上面跑。

Repo License

Your approach seems very useful. I would like to fork it, but I couldn't find its code license. Seems you have not included any to your repository. I appreciate if you add the license file to it.

Thanks

predict的一些问题

你好，我试了一下predict跑了一张图片，但是跑出来的效果很差，我用的是gen_model_epoch_160.pth 然后config里面都是默认的。
原图

预测图

我的图片大小都是640x480，但是我看数据加载的时候加载出来是(3,480,640)，而且config里面设置的是宽480高640？这里是不是有问题啊

predict.py issue

Traceback (most recent call last):
  File "predict.py", line 84, in <module>
    predict(config, args)
  File "predict.py", line 58, in predict
    allim[0, 2, :] = heat_att
IndexError: index 2 is out of bounds for axis 1 with size 1

This error occurred and ignored code 'allim[0, 2, :] = heat_att' and executed successfully.
Is there any problem with the performance of the model?

predict.py error

Running predict.py leads to an error:

python predict.py
Traceback (most recent call last):
  File "predict.py", line 13, in <module>
    from untitled1 import Generator
ModuleNotFoundError: No module named 'untitled1'

Why use bg-shadow instead of Mask file provided by ISTD?

Hi,
I noticed that the original train_B folder is not used, and TrainDataset simply returns "np.clip((t-x).sum(axis=2), 0, 1)" as shadow mask(or matte). Does this rough mask perform better than original mask, or are there other reasons for using this setting? As it's a known issue that a part of images in ISTD have highly non-consistent lightness between bg & shadow.

SSIM data_range

File "/usr/local/lib/python3.8/dist-packages/skimage/metrics/_structural_similarity.py", line 194, in structural_similarity
raise ValueError(
ValueError: Since image dtype is floating point, you must specify the data_range parameter. Please read the documentation carefully (including the note). It is recommended that you always specify the data_range anyway. 作者您好 我用SSIM这个缺少一个data_range参数 按照您给的640 480 像素 这个参数值应该是写多少呢

untitled1 not found 问题依然存在！请问是什么原因

Mis-Match Results

Hello Brother,
Thanks for the amazing work.

I have done inference of ISTD test data set images with the the checkpoint provided by you. I am unable to get the good results.
The model is performing extremally well when there is a "grey" color image at its input.

But when there is a combination of colors i.e. even 2 colors in an image, there is hardly any difference between the input image and the output shadow image produced. Few of the results are attached.

TRAINING ISSUE

I am trying to train on ISTD dataset and facing the following error. All the images in train, test dataset are of 640x480.
The error arises after 1 epoch.

Traceback (most recent call last):
File "train.py", line 184, in
train(config)
File "train.py", line 159, in train
log_validation = test(config, validation_data_loader, gen, criterionMSE, epoch)
File "/content/drive/MyDrive/THESISMODELS/SpA/SpA-Former-shadow-removal/eval.py", line 49, in test
allim[0, 0, :] = in_rgb * 255
ValueError: could not broadcast input array from shape (3,640,480) into shape (3,256,256)

ModuleNotFoundError: No module named 'attrdict'

Hi!
First of all, thank you for this repo as it's really interesting. I tried to clone and run it but it seems it doesn't have the module named 'attrdict'.