rq-wu/ridcp_dehazing

[CVPR 2023] | RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors

🔥 RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors (CVPR2023)

This is the official PyTorch codes for the paper.

RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors
Ruiqi Wu, Zhengpeng Duan, Chunle Guo^*, Zhi Chai, Chongyi Li （ * indicates corresponding author)
The IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR), 2023

[Arxiv Paper] [中文版 (TBD)] [Website Page] [Dataset (pwd:qqqo)]

🚀 Highlights:

• SOTA performance: The proposed RIDCP achieves superior performance in both qualitative and quantitative comparisons with SOTA methods.
• Adjustable: Our RIDCP can adjust the degree of enhancement manually.

📄 Todo-list

• Release code for VQGAN pre-training and CHM weights acquiring
• Website page
• Colab demo

Demo

Video examples

       

Dependencies and Installation

• Ubuntu >= 18.04
• CUDA >= 11.0
• Other required packages in requirements.txt

# git clone this repository
git clone https://github.com/RQ-Wu/RIDCP.git
cd RIDCP

# create new anaconda env
conda create -n ridcp python=3.8
conda activate ridcp 

# install python dependencies
pip install -r requirements.txt
BASICSR_EXT=True python setup.py develop

Get Started

Prepare pretrained models & dataset

1. Downloading pretrained checkpoints

Model	Description	🔗 Download Links
HQPs	VQGAN pretrained on high-quality data.	[Google Drive (TBD)] [Baidu Disk (pwd: huea)]
RIDCP	Dehazing network trained on data generated by our pipeline.
CHM	Weight for adjusting controllable HQPs matching.

2. Preparing data for training

Dataset	Description	🔗 Download Links
rgb_500	500 clear RGB images as the input of our phenomenological degradation pipeline	[Google Drive (TBD)] [Baidu Disk (pwd: qqqo)]
depth_500	Corresponding depth map generated by RA-Depth(https://github.com/hmhemu/RA-Depth).
Flick2K, DIV2K	High-quality data for VQGAN pre-training	-

3. The final directory structure will be arranged as:

datasets
    |- clear_images_no_haze_no_dark_500
        |- xxx.jpg
        |- ...
    |- depth_500
        |- xxx.npy
        |- ...
    |- Flickr2K
    |- DIV2K

pretrained_models
    |- pretrained_HQPs.pth
    |- pretrained_RIDCP.pth
    |- weight_for_matching_dehazing_Flickr.pth

Quick demo

Run demos to process the images in dir ./examples/ by following commands:

python inference_ridcp.py -i examples -w pretrained_models/pretrained_RIDCP.pth -o results --use weight --alpha -21.25

Train RIDCP

Step 1: Pretrain a VQGAN on high-quality dataset

TBD

Step 2: Train our RIDCP

CUDA_VISIBLE_DEVICES=X,X,X,X python basicsr/train.py --opt options/RIDCP.yml

Step3: Adjust our RIDCP

TBD

Citation

If you find our repo useful for your research, please cite us:

@inproceedings{wu2023ridcp,
    title={RIDCP: Revitalizing Real Image Dehazing via High-Quality Codebook Priors},
    author={Wu, Ruiqi and Duan, Zhengpeng and Guo, Chunle and Chai, Zhi and Li, Chongyi},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2023}
}

License

Licensed under a Creative Commons Attribution-NonCommercial 4.0 International for Non-commercial use only. Any commercial use should get formal permission first.

Acknowledgement

This repository is maintained by Ruiqi Wu.

ridcp_dehazing's Issues

关于生成有雾图像

您好，感谢您如此出色的工作，在阅读完您的论文后我也深受启发。

在阅读合成有雾图像的代码时，我发现transmision map的生成用了以下代码：
t = np.exp(-(1- img_depth) * 2.0 * beta)
所以想请教一下这里为什么需要将图像深度img_depth变为(1- img_depth) * 2.0？

期待您的回复。

关于模型测试

您好，在论文中您将RIDCP和其他的去雾模型(e.g. MSBDN, D4)在RTTS上进行了比较，请问其他去雾模型的训练方式是怎样的呢，是在RESIDE的合成数据集上进行训练的吗？另外您能提供一下计算FADE, BRISQUE和NIMA的代码吗？

不胜感激！

CHM训练

想问一下第三步什么时候上传

offset abs

我在训练第二步时总会出现 Offset abs mean is 53.02199172973633, larger than 50.这种情况。

quick demo

当我使用您readme中的 python inference_ridcp.py -i examples -w pretrained_models/pretrained_RIDCP.pth -o results --use weight --alpha -21.25这段命令时会出现no such file pretrained_models/CHM_weight.pth 这个错误，而我修改成您所提供的pretrained_RIDCP.pth后就出现了'dict' object has no attribute 'detach'此错误，或许您能帮我解答一下出现此问题的原因吗？

关于Stage I 的预训练

非常感谢您的工作。请问可否分享一下 Stage I 的预训练代码或经验呢？最近在用自己的数据集训练 Stage I，但效果不太理想。
期待您的回复！

KeyError: "No object named 'HazeOnlineDataset' found in 'dataset' registry!"

关于视频去雾

非常感谢您的工作！
请问有端到端视频去雾的实现脚本吗？

关于CHM和VQGAN训练的开源

您好！请问近期会开源CHM和VQGAN的训练代码、config嘛，将会很有帮助，多谢！

CUDA out of memory

你好，感谢你的贡献。我正在尝试复现，但是用2080Ti提示显存不够，换3090就要装torch 1.10以上的了，装完又提示我basicsr需要1.8的torch。我有点没办法了 :(

关于训练模型

您好！我想将您的模型在其他图像修复场景里面重新训练或者finetune，我看到readme里面train部分还是TBD，请问是训练超参数还未公布吗？或者是否可以简单介绍一下如何使用basicsr/train.py训练。谢谢！

模型训练问题

你好，我在自己的数据上重新训练的时候出现了以下错误，请问应该如何解决，谢谢！
0%| | 0/500 [00:00<?, ?image/s]Traceback (most recent call last):
File "basicsr/train.py", line 243, in
train_pipeline(root_path)
File "basicsr/train.py", line 236, in train_pipeline
model.validation(val_loader, current_iter, tb_logger, opt['val']['save_img'])
File "/home/qyj/DCP/basicsr/models/base_model.py", line 48, in validation
self.nondist_validation(dataloader, current_iter, tb_logger, save_img, save_as_dir)
File "/home/qyj/DCP/basicsr/models/dehaze_vq_model.py", line 266, in nondist_validation
sr_img = tensor2img(self.output)
File "/home/qyj/DCP/basicsr/utils/img_util.py", line 60, in tensor2img
raise TypeError(f'tensor or list of tensors expected, got {type(tensor)}')
TypeError: tensor or list of tensors expected, got <class 'tuple'>

Confusion arises regarding some details of the paper.

(1) It is unclear how to calcuate the Code activation frequencies in Figure 4.

(2) The paper mentions using a binary search algorithm to iteratively find the approximate optimal solution for αˆ, but it is not clear how to obtain αˆ with this algorithm.

我的一些看法

。。。

About training loss

注意到您在论文中提到了第二阶段的encoder和decoder的对应loss，在代码中是用一个总的loss和一个optimizer进行更新的，不知道您是否用过两个optimizer分别对encoder和decoder进行更新呢？

模拟雾天的代码

你好，请问这个模拟雾的代码可以公开吗，谢谢！

About Model Efficiency

I did the efficiency test of image dehazing, and the model only got 1.5fps or even lower. Did you test it when doing the experiment?

真实场景的雾天数据合成

近期在网络上收集并看了些图像去雾的代码与论文，发现现今大多数论文多是在刷现有数据集的PSNR或SSIM分数。
而现有数据集多是用去雾机制作的成对数据集或者是利用RGBD数据中D参数合成雾天数据，基本很少有论文真正在真实雾天数据上做的算法。
因为在现实场景中不易构建成对数据集，退化模型也不是完全拟合现实场景。

关于合成数据，我搜集到以下方案：

img = cv2.imread("./1.jpg")  # 需要处理的文件夹
mask_img = img.copy()
# mask_img[:, :] = (144, 128, 112)  # 112, 128, 144 RGB 三种颜色相同时为灰度色彩
mask_img[:, :] = (166, 178, 180)  # 雾的颜色可以调整，看看哪个效果符合自身的需要
A= 0.2 # A 主要控制雾的浓度
image = cv2.addWeighted(img, round(A, 2), mask_img, 1, 0) 
cv2.imwrite("./res0.png", image)

光学模型合成雾

img = cv2.imread("./test.png")
img_f = img / 255.0  # 归一化
(row, col, chs) = img.shape
A = 0.5  # 亮度 AA = [0.8 0.85 0.9 0.95 1]
beta = 0.000005  # 雾的浓度 #  [0.1 0.2 0.04 0.06 0.08] ;
beta = 0.005  # 雾的浓度 最高
size = math.sqrt(max(row, col))  # 雾化尺寸，可根据自己的条件进行调节，一般的范围在中心点位置但不是很大，可自己手动设置参数
center = (row // 2, col // 2)  # 雾化中心 就是图片的中心
for j in range(row):
    for l in range(col):
        d = -0.04 * math.sqrt((j - center[0]) ** 2 + (l - center[1]) ** 2) + size
        td = math.exp(-beta * d)
        img_f[j][l][:] = img_f[j][l][:] * td + A * (1 - td)  # 标准光学模型，图片的RGB三通道进行加雾
cv2.imwrite('./dst.png', img_f * 255)

请问还有其他比较好的方案，能够利用现实图合成雾图，最终用于去雾吗

Great work! I would like to ask a simple question

Great idea, and the dehazing effect is also quite good. Inspired by RestoreFormer, I tried using VQGAN to achieve dehazing, but the actual results were not satisfactory, mainly because the reconstructed images were blurry. Your approach to solving the problem has inspired me, and it's excellent work.

I think I still need to ask you a question. In your proposed method, is the new synthesis pipeline more important or the model based on VQGAN? I saw a few test samples in the paper where the baseline models were trained using the new dataset, but I'm curious about the specific quantitative results (even though I know that NR-IQA metrics are not very reliable).

how to test it on my own video？

Hello~ Thank you very much for your work, and i want to know how to test it on my own video？
i tried it ‘python inference_ridcp.py -i v1.mp4 -w pretrained_models/pretrained_RIDCP.pth -o results --use_weight --alpha -21.25’，but i got the error：

模型部署问题

您好，我用您的预训练权重对现实中的场景图像进行了测试，效果不错。
请问能否提供下onnx，engine模型量化的代码和对应的C++的部署代码呢？
谢谢~

复现问题

你是通过合成数据的psnr结果确定的迭代次数嘛，我总是复现不到你给模型权重的效果，总是差一些。是bachsize的大小的原因么，还是迭代次数或者什么原因

Google Drive links

Hello,

Thank you for your contribution.
The google drive links for the pretrained models and the dataset do not work for me.
They point back to your github page.
Could you make an update please?

Thank you very much,
Maria

关于您公布的500张数据集

感谢您的工作！
有一个小问题，您对比的方法大部分是基于RESIDE数据集来训练的，直接用他们的pretrained model和您的model对你的话，是不是不算太严谨，就是是否考虑过用同样的gt（比如RESIDE）来基于您提到的合成pipeline进行在线合成训练 来进行对比。

还想请问下您公布的500张数据集如果换成其他真实干净图像，再基于您的数据合成pipeline进行训练，是否会对结果有影响？

Dataset

I need this dataset but the link is not working. please give me access to dataset.

运行Demo的出错疑问，往有空解惑！

终端运行BASICSR_EXT=True python setup.py develop 提示BASICSR_EXT命令不能识别；
使用demo运行语句，提示 NameError: name 'deform_conv_ext' is not defined

ITS训练集

您好，ITS数据集里同一张清晰图对应有10个雾度的有雾图片，请问是10个雾度图都训练呢还是一张清晰图挑一个雾度图就好

cannot find the CHM pretrained weight

cannot find the CHM pretrained weight ,能否提供一下