huizeng / image-adaptive-3dlut Goto Github PK
View Code? Open in Web Editor NEWLearning Image-adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time
License: Apache License 2.0
Learning Image-adaptive 3D Lookup Tables for High Performance Photo Enhancement in Real-time
License: Apache License 2.0
Hey, thanks for your great work!
I 've trained the model using the unpaired version of code and got great results. But I found the loss of D was quiet abnormal: D always give the fake and real input very close scores(at least same signal), no matter how many epochs we train, even when the 800 epochs meet the end and visualization of the result was quiet good. I wonder how could the result get that good with such a weak D and unbalance scores between G&D?
torch.utils.ffi is is deprecated. Please use cpp extensions instead
torch 1.x can not build successfully, could you provide a modefied version using torch.utils.cpp_extension ?
您好,请问FiveK数据集中的DNG图像,是如何转换成JPG图像的?使用了什么软件,需要设置什么参数,这些的信息可以提供一下吗?感谢~
from ._trilinear import lib as _lib, ffi as _ffi
ImportError: Image-Adaptive-3DLUT/trilinear/_ext/trilinear/_trilinear.so: undefined symbol: __cudaRegisterFatBinaryEnd
Hi, I run your code with the following environment:
I try the default paired setting without changing any hyper-parameters, but I can only get 23.67 PSNR, 0.897 SSIM and 9.19 lab
on the fiveK test-set. I also train the same settings several times and get the following results:
Model | PSNR | Epoch |
---|---|---|
baseline_0 | 23.107576 | 121 |
baseline_1 | 22.001934 | 236 |
baseline_2 | 18.170982 | 358 |
baseline_3 | 23.083334 | 332 |
Note that the 23.67 PSNR
result is obtained using the model baseline_0
and epoch 121
.
From the results, I find that the performance is not stable. And I want to know how to reproduce the result in the paper.
Thank you very much.
Hi, thanks for your great work.
I try your code on fiveK dataset and find that it can run successfully using the default setting (batch size=1). However, when I switch to a larger batch size (e.g. 2 or 16), an error occurs as follows:
RuntimeError: stack expects each tensor to be equal size, but got [3, 507, 472] at entry 0 and [3, 408, 482] at entry 1
From my previous experience, maybe you need to align the sizes of different images then put them into a tensor.
I'm very interested in your work! Especially, how does it work when local tone mapping is applied. Looking forward!
hello, how can i solve this problem?
I looked it up online and said it was memory or Pointers?
Thank you for your efforts . I have test a single image folder in demo_eval.py and ouuputs a satisfactory image! Now I just to know how to test a video or a image folder in demo_eval.py! Looking forward to your reply ! Have a good day!
train_input.txt和train_label.txt中,图像列表是随机划分的吗?
为什么要分成input和label呀,有不同的作用吗?
Hi, I am trying to replace the matlab evaluation script with python style for some reason. Then I do something like this:
from skimage.metrics import structural_similarity as img_ssim
# To match the implementation of Wang et. al. [1]_, set `gaussian_weights` to True, `sigma` to 1.5, and `use_sample_covariance` to False.
ssim_val = img_ssim(test_img, gt_img, multichannel=True, data_range=255.0, gaussian_weights=True, sigma=1.5, use_sample_covariance=False)
However, the results are as follows ('for a4994.png'):
Could you please tell me where is the difference between the implementations of the two versions? Or can you provide a python script for evaluating SSIM and Lab?
Thank you very much.
Hi, I was wondering how you generated the training images? For the sRGB input, I tried 'dcraw -T', 'dcraw -w -T' and 'dcraw -w -M -T', but I can't get the same image as yours.
Also, how did you get the XYZ 16bit inputs? I tried using you sRGB input, degamma, convert to XYZ, gamma, and save as 16bit using openCV, but the result is still different from your XYZ input.
Thanks.
Hi there,
In TrilinearBackwardCpu
, I saw only lut_grad
is well calculated, and image_grad
is considered as identity.
In this artical, the gradient of image also needs to be backward passed to the CNN weight generation layers. A more correct graident calculation can make this project more elegant.
Yours,
Pytorch version: 1.5.1, cuda: 9.0
Error follows: AttributeError: module 'trilinear_cpp.trilinear' has no attribute 'trilinear_forward_cuda'
So you have to modify
https://github.com/HuiZeng/Image-Adaptive-3DLUT/blob/master/trilinear_cpp/src/trilinear_cuda.cpp#L32
to
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
m.def("trilinear_forward_cuda", &trilinear_forward_cuda, "Trilinear forward");
m.def("trilinear_backward_cuda", &trilinear_backward_cuda, "Trilinear backward");
}
Finally, I can run it on the demo_images, the results is very good.
envs: trilinear_cpp. pytorch 1.6
Is the unpaired pretrained model mismatched ?
RuntimeError: Error(s) in loading state_dict for Classifier:
Missing key(s) in state_dict: "model.7.weight", "model.7.bias", "model.9.weight", "model.9.bias", "model.13.weight", "model.13.bias", "model.16.weight", "model.16.bias".
Unexpected key(s) in state_dict: "model.8.weight", "model.8.bias".
size mismatch for model.6.weight: copying a param with shape torch.Size([64, 32, 3, 3]) from checkpoint, the shape in current model is torch.Size([32]).
size mismatch for model.6.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([32]).
size mismatch for model.10.weight: copying a param with shape torch.Size([128, 128, 3, 3]) from checkpoint, the shape in current model is torch.Size([128, 64, 3, 3]).
size mismatch for model.12.weight: copying a param with shape torch.Size([3, 128, 8, 8]) from checkpoint, the shape in current model is torch.Size([128]).
size mismatch for model.12.bias: copying a param with shape torch.Size([3]) from checkpoint, the shape in current model is torch.Size([128]).
Hi, very honor to see your works
Actually I have trained with my custom dataset, but there are some issues I cannot verify with my knowledge.
So I am trying to analyze the shapes(?) of LUTs
Is there any tool to visualize the LUTs which can show the results just like in your paper?
Thank you for the answer
Hi, thanks again for the great work! I just found a bug in file demo_eval.py. OpenCV reads image into BGR, so after line 77, i think image should be converted to RGB before proceeding.
Please let me know if i'm right.
Thanks again for the great work!
In some situation, training code raises a device error:
cuda error: an illegal memory access was encountered
After debug I found that the main reason is Tensor
used during the training is not on the same device. For example:
In Generator3DLUT_identity
and Generator3DLUT_zero
, self.LUT.device
is cpu
. In TrilinearInterpolationFunction
, int_package
and float_package
are also on cpu
. However, the input and the output of the network are cuda tensor, causing sometimes device error occurs when running the model.
To solve the problem, it's better to init all tensors as the same and dynamic type, instead of initializing the tensor on fixed devices.
I downloaded the pretrained model to test and found that it only provides access to first three LUTs. In the eval code, it is indicated that a full test should use 5 LUTs. I wonder if other pretrained LUTs will be released for me to test the performance of your work to a complete extent?
Thanks!
Hello, your work is very good. How to solve this problem? Do you need to add this function trilinear_func.py? Can you update this function to your code in the github?
您好,请问您的这篇文章已经被TPAMI2020接受了吗?因为对论文内容很感兴趣,所以有点好奇工作是否被接受,谢谢您的解答!
Hello, If this work is admit by TPAMI 2020? I'm very interested in this paper, thanks for you reply!
您好,我将代码改成单卡多batchsize进行训练,psnr值反而没有batchsize为1时效果好,请问这个是什么原因导致的呢,如果是多GPU训练,每张GPU上batchsize为1,最终效果是否也没有单卡单batchsize效果好呢,
Hi, thanks again for the great work! I was trying to mimic the result of imaging pipeline enhancement. I checked the HDR+ dataset but I can't find the aligned Nexus 6p subset you mentioned in the paper. Can you please give me a hint on the names of HDR+ Dataset?
Thanks.
作者你好!我想问一下你运行项目所使用的cuda版本,使用 python3 image_adaptive_lut_evaluation.py后,总是出现问题,我想可能是环境导致的,,望回复,感谢!问题如下:
/root/anaconda3/envs/pytorch041_2/lib/python3.6/site-packages/torch/nn/modules/upsampling.py:122: UserWarning: nn.Upsampling is deprecated. Use nn.functional.interpolate instead.
warnings.warn("nn.Upsampling is deprecated. Use nn.functional.interpolate instead.")
/root/anaconda3/envs/pytorch041_2/lib/python3.6/site-packages/torch/nn/functional.py:1961: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
"See the documentation of nn.Upsample for details.".format(mode))
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Good day, Dr. Zeng,
Thanks for your great work. Here I have some questions about UIE [11] as cited in your paper.
Please see if I miss something in your arguments. All in all, the above won't change much about your work's value. However, the discussion here is to view things more objectively and correctly.
Thanks for your attention :-)
Hi, thanks for the great work! Can you please share the unpaired pretrained models? Thanks!
Hi, I want to train the model. I have downloaded dataset from the link you provided, but I am not sure where is train_label.txt and train_input.txt, which is used in datasets.py. Can you provide an example of these two files?
Thanks for your great work.
My pytorch version is 1.6.0, I build the project by ‘cd trilinear_cpp, sh setup.sh’, and then I run 'python demo_eval.py' command and get the enhanced image of a1629.jpg. But I find the 'red' flowers becomes 'blue' flowers, and the ‘blue’ tail of the bird becomes ‘brown’ tail, it's different from the result provided in your paper. Is there something wrong in the model you provided here or is there something else I should do before I run 'python demo_eval.py' ? The following picture is the result I get.
Look forward your reply.
Hi there,
This is probably more a modelling question rather than an issue with the code, but it might be worth discussing here anyway.
I am training your model and found that trained LUTs can actually map normalized RGB to values outside the range [0,1]. Don't you think this makes little sense?
PyTorch: 1.6, CUDA: 10.0
According to the README, modified the code, and also modified the setup.sh
toexport CUDA_HOME=/usr/local/cuda-10.0
.
But when I try to run demo_eval.py
, I meet the error:
cudaCheckError() failed : invalid device function
Process finished with exit code 139
Does anyone knows how to solve that? I have no idea after search on Google.
return {"A_input": img_input, "A_exptC": img_exptC, "B_exptC": img2, "input_name": img_name}
作者,我想了解再unpaired形式的训练中的img2 的作用是什么, 我看到在fiveK和hdrplus数据集中都用到了这个参数,一直没想明白,望解答,thank you very much
Thank you for your efforts. Under the link of the dataset, the download dataset is incomplete or the download fails. I want to confirm whether the dataset path has changed? Could you please attach a link to Baidu Cloud? Thank you very much!
Hi, the provided local tone mapping code is written in MATLAB
%% Finally, shift, scale, and gamma correct the result gamma = 1.0/1.0; bias = -min(OUT(:)); gain = 0.8; OUT = (gain*(OUT + bias)).^gamma; % figure imshow(OUT); imwrite(OUT,'a1509_T.jpg');
I found that the range of OUT is not in [0, 1], when I convert this script to Python, the result is not correct when using min-max normalization, do you know how to normalize it in python?
Hi,
I tested it successfully with the original image(a duck eating grass).
But when I replace it with another jpg image, an error occurred:
Traceback (most recent call last):
File "demo_eval.py", line 86, in
LUT = generate_LUT(img)
File "demo_eval.py", line 66, in generate_LUT
pred = classifier(img).squeeze()
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/dockerdata/ruodai1/Image-Adaptive-3DLUT/models_x.py", line 99, in forward
return self.model(img_input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
input = module(input)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 419, in forward
return self._conv_forward(input, self.weight)
File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 416, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [16, 3, 3, 3], expected input[1, 4, 256, 256] to have 3 channels, but got 4 channels instead
I use the snipper in windows to get this image and save it in jpg format. Is there anything wrong? Thanks!
您好,我想处理高分辨率的图片6000x4000,想请教下,加大3Dlut的维度(比如256),或者采用更大的分类骨干网络能否提高效果呢,您能提供一些建议吗,非常感谢
您好,麻烦请教下,我现在想做图像增强任务:一张图片要经过色温调整,对比度调整,颜色调整三种递进的流程,要输出三张不同调整效果图,我想在您的网络结构下将这三个处理过程串行在一起,分阶段进行监督(LUT有三组),我这边有的数据对是一张图片对应三张效果图,这样的流程可行吗
Thanks.
In case of weight_r (deleted g or b statement for ease):
self.weight_r = torch.ones(3,dim,dim,dim-1, dtype=torch.float)
self.weight_r[:,:,:,(0,dim-2)] *= 2.0
def forward(self, LUT):
dif_r = LUT.LUT[:,:,:,:-1] - LUT.LUT[:,:,:,1:]
tv = torch.mean(torch.mul((dif_r ** 2),self.weight_r)) + torch.mean(torch.mul((dif_g ** 2),self.weight_g)) + torch.mean(torch.mul((dif_b ** 2),self.weight_b))
Here, 1) I wonder why do we multiply dif_r ** 2 and self.weight_r instead of using just dif_r ** 2?
and 2) why do we multiply the edge elements of weight_r by 2.0?
I really appreciate to have this work :)
As far as I know, the Input setting used in the DeepUPE experiment is (default), and I downloaded the sample you provided, I found that the Input used in your experiment is the fourth (highlighted).
The default input setting is darker than the fourth input setting, which may be the reason why your method SSIM is particularly high.
In addition, your train/test split setting of FIVEK dataset is also inconsistent with DeepUPE. Considering that DeepUPE does not share the training code, I would like to know whether you have re-implemented and trained the DeepUPE model?
Hi,
Thank you for sharing the clear codes. The idea is really nice. However, when trying to run your code on PyTorch 1.7, I get error message telling me that autograd function of TrilinearInterpolation
is deprecated. Any idea on how to fix this?
Thanks!
Sorry, I am a newbie for this part
I am having a difficulty with building the package 'trilinear'
I have followed the guide in README but it doesn't work with the message 'ImportError: libc10.so: cannot open shared object file'
Do you have any idea of this phenomenon?
I am using following environment
Python 3.7.6
torch==1.7.0
torchvision==0.8.1
numpy==1.19.4
opencv-python==4.4.0
nvcc v10.0
GPU: Tesla K80
Any comments are welcomed .. Thank you
I'm very interested in your work! But, When I read the paper, I found a problem why is the PSNR value of UPE much lower than that of HDRNet in your paper. As shown in the figure below:
However, The paper on Underexposed Photo Enhancement using Deep Illumination Estimation Indicates that their model performance is higher than HDRNet
您好,运行image_adaptive_lut_evaluation.py代码的时候,环境是按照你的要求配置的,但是报错RuntimeError: PyTorch was compiled without NumPy support,网上的解决方法不管用;
另外想问一下,这个代码支持多卡运行吗?
很感谢您分享的代码,期待回复哈祝好
Could you please specify the license for the repo?
Thanks
BR
IG
1)Given an input image of any resolution, you simply use bilinear interpolation to downsample it to 256 x 256。Why did you put this step in the network instead of image preprocessing? Is this equivalent?
2)If I don’t want to use InstanceNorm, are there any suggestions for modifying the architecture of the CNN model?
3)Since bilinear interpolation is time-consuming, will it affect the result if I change it to nearest neighbor?
Looking forward to your reply!
For example, lut[33][33][33] ={0.8224368, 0.82099277, 0.8277654}.
The color of colorless issues will be changed after the 3DLUT operation。
The diagonal r:g: b of the cube should be 1:1:1
is that possible to replace your TrilinearInterpolation with torch.nn.functional.grid_sample? If ok, it would be more convenient.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.