cchen156/learning-to-see-in-the-dark

Learning to See in the Dark. CVPR 2018

learning-to-see-in-the-dark's Issues

Result Image(Sony Model -The green is scattered)

I use your sony pretrainned model, Execute test_sony.py
I wonder why green dots are spreading in the output image.

iphone 6s weights

The official website contains a low light image and its result when run on SID model (http://web.engr.illinois.edu/~cchen156/SID/examples/6s.html). I could not find the weights for this in the repository. Is it possible to upload the weights for when trained on iPhone 6s?

apply whitebalance before feed into network

@cchen156 Hi, congrats first. In your training settings now, you aim to learn white balance within the network. Have you tried to apply white balance on input(those short exposure images) before feeding them into networks? If so, how does the result look like? THX!

what if we only have rgb input images

If we only have rgb input images, how should I retrain the model and test it?
Thanks

[Enhancement] Add input image to the result

Hi, thanks for your great work.
This issue is just a suggestion of possible enhancement.
When testing the model, currently the output images are ground truth, the output from the neural network, and the input scaled up to the same mean as ground truth.

My suggestion is to add

scipy.misc.toimage(input_full[0,:,:,:]*255,  high=255, low=0, cmin=0, cmax=255).save(result_dir + 'final/%5d_00_%d_in.png'%(test_id,ratio))

to output the initial input image, to show how unusable the original photo is.

Python 3 support

I noticed that the README suggests it needs Python 2.7 but from looking at the libraries, nothing jumps out that would be dependent on 2.7, so I'd have expected it to run with 3.

Is it simply that 2.7 is what you happened to develop / test with, or is there something in your code that specifically needs 2.7?

can't download data

why can not i download the data?
The website always return:
Bad Request
Error 400

misalignment problem in the dataset

A question in the test_Sony.py is that, the scipy.io is really slow and cv2.imwrite is really fast. In the 8-bit saving way, do this two methods inference the finnaly PSNR & SSIM?

About PSNR

Hi @cchen156 i test psnr of ur model on test samples 1-49 in ur test_list.txt , and the psnr is ~32dB ,is that the same with urs ? i didnt change any of ur codes
My PSNR Calculation ：
PSNR（img1,img2）:
mse = np.mean((img1 - img2)^2)
return 10*np.log10(255^2 / mse);

Questions regarding to the PSNR/SSIM results

Hi @cchen156 ,

This work is very exciting. But I have a question about the quantitative results reported in the paper Table3.

I downloaded the source code and trained model, and tested Sony dataset. I generate GT by Rawpy and apply Gamma transfer after it, since you apply Gamma transfer on the output of the network in your test code. I found that the results looks fine, but the average PSNR/SSIM in the testset is 25.44/0.713, which is quite lower than 28.88/0.787 reported in Table3 in your paper.

If I have made some mistake? How can I get the reported results.

Thanks,
Wenbo

int pack_image the 'subtract the black level and scale the data' :why is im = np.maximum(im - 512, 0) / (16383 - 512)

why black level equal 512 and why is 16383 ,What is meaning of the number 16383?

amplification ratio problem in Sony dataset

Evaluation network？

Is there any evaluation network?

The comparison with HDRnet

Hi,
Thanks for sharing this work and the result is awesome ! @cchen156
I find comparison with hdrnet in the project page ， so are you using HDRnet prettrained model from @mgharbi 's repo or you trained the comparison model youself？

I want to reproduce the comparison but seems no night related pretrained model is found for hdrnet， would you like to share the comparison pretrained model ? Thanks

Result of sony

I downloaded the sony data of 22 GB, the training and testing programs were executed, the folder named 'result_sony' is created, with some files in it, but those files are not executable and there were no images to be seen, can you please tell me that was result was expected once the program is executed completely.

How about training with 8-bit jpg image?

I just tried to train Sony dataset after converting the raw data into jpg image and found the result quite terrible. So are there any tricks when training 8bit data?
P.S.:
The channel number of input is 12, which results from packing each channel of rgb into four.

Trained model (for iPhone)

Amazing results, congrats!

Is it possible to download from somewhere the pre-trained model?

Thanks a lot!

The gpu memory is occupied when I run the train_Sony.py

The gpu memory is occupied when I run the train_Sony.py, so I set

tfconfig = tf.ConfigProto(allow_soft_placement=True) tfconfig.gpu_options.allow_growth=True sess = tf.Session(config=tfconfig)

But there is still the question...?

how to export the raw data of iphone?

GREAT WORK!

There's a certain app you mentioned in your paper, which can set parameters and export raw data.
Could you please tell me what's the name of the app?

I only know we can access YUV data through Apple's API.

Thanks~!

ratio multiplication

@kkruups @prabodhw96 @cchen156 Hi, congrats for your work. In your paper and code, during training, input and corresponding ground truth are first scaled between [0-1], and then input are scaled by ratio but the ground truth is still between [0-1]. Have you tried not to scale input by ratio and keep input and gt both in range [0-1] to train the network? Will it get better result and what's your opinion? THX!

Any hint for running on other camera raw (nikon,canon)

Hi!
I was able to run the examples for Sony raw data and the results are amazing.
I would like to experiment with Canon and Nikon raws but with Canon's raws the model seems to never converge. I am able to run it but i don't get any output. I modified the test_Sony file changing the ARW extension in CR2 (Canon's raw). Any advice?

Thank you!

chind mainland can not download the models

when I run the download_models.py
there is no response. can i download the models in other ways?

freeze model

Can you get the inference freeze.pb file for your model?
what is your input node name ? output node name?

Training time

How long did the training on Titan X take? What was the final loss?

Download_models::File not Found!!

download_models.py

Any script for downloading Sony 12GB?

Hi Chen,

Thanks for your code, it is very useful!

I just wonder could you please provide the script for downloading Sony 12GB? It will help me a lot!

Thanks,

upsample_and_cancat(x1,x2,output_channels,input_channels) does not contant the Crop operation

@cchen156 Do you have the "Croplayer" crop operation in this fuctions,
It only have 'DECONVOLUTION' ans 'CONCAT'，

New dataset download links

I downloaded the two zip files for myself, and with my host (wasabi) I pay for storage but not bandwidth, so I'm willing to share the links to the datasets here:

https://s3.wasabisys.com/datasets/Learning-to-See-in-the-Dark/Fuji.zip
https://s3.wasabisys.com/datasets/Learning-to-See-in-the-Dark/Sony.zip

Feel free to use them, or get back in touch if you want me to stop making them public. I'll be keeping them up for the foreseeable future.

Suggestions on speed up the training procedure

comparison between output of SID(see in the dark) and photoshop

I tried to compare the output generated by SID (O-sid) and photoshop (O-ps, with all auto from DNG to PNG), and according to what I saw:
1/ when the light is extremely low, O-sid is better than O-ps.
2/ but when the light is low and not extremely, O-ps is better than O-sid.

Also, O-sid loses quite a few texture compared to O-ps.
I wonder is this as expected?

train_Sony.py refer to the wrong dir

I think that train_Sony.py at line 129

allfolders = glob.glob('./result/*0')

and train_Sony.py at line 136

if os.path.isdir("result/%04d" % epoch):

are wrong.

The correct programs are

allfolders = glob.glob(result_dir + '*0')
if os.path.isdir(result_dir + "%04d" % epoch):

, right?

error in test_Sony.py

after download and extract raw Sony data in dir dataset/Sony.
run test_Sony.py
got error at line 151:
im = raw.postprocess(use_camera_wb=True, half_size=False, no_auto_bright=True, output_bps=16)

rawpy._rawpy.LibRawFatalError: b'Unsufficient memory'
could you have met this?
Appreciate for answer.

Pnsr and SSIM

Hello,@cchen156,I used the cpkt file you provided to test PNSR and SSIM, but the measured value is still lower than the original text, PNSR=28.639, SSIM=0.7682, is it different from the method I used to test pnsr and ssim? Can you provide the method or code for testing pnsr and ssim?

16bit bayer filter

1. Can I use a Sony network model for a 16bit bayer filter?
2. Is it possible to share iphone 6s raw images(dng format) in paper?

TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

What`s this probably mean?Here is the log:

Traceback (most recent call last):
File "test_Sony.py", line 117, in
out_image = network(in_image)
File "test_Sony.py", line 76, in network
up6 = upsample_and_concat(conv5, conv4, 256, 512)
File "test_Sony.py", line 50, in upsample_and_concat
deconv_output = tf.concat([deconv, x2], 3)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1075, in concat
dtype=dtypes.int32).get_shape(
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 669, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 176, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 165, in constant
tensor_util.make_tensor_proto(value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 367, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/tensor_util.py", line 302, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).name))
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

The trained model can't be downloaded

The download_models.py always got a HTTP connection error, could you please provide a available download links? Thanks.

how to use the groundtruth data on the disk

I don't have enough RAM，can you give me some suggestions to revise the code. Thanks for a lot.

Why the batch_size is 1?

I don't understand why you set the batch_size=1, i.e., only 1 picture will be trained in one iteration. Have you ever tried other batch_sizes?

The “dataset” cannot be downloaded

Hi:
I found I cannot download the dataset, could you pls give another URL?
https://docs.google.com/uc?export=download

Bad Request
Error 400

Pedestrian detection in low light

Hi, I am work on the project which can detect and track pedestrian in low light or night. Then how can I use your model for my dataset?

How can I test the model on my own dataset?

The model is nice, but how can I use the model to convert my own single jpg(or png) img ? Is there any Introduction?

Why psnr values differ so much when input Sony and Fuji rgb datasets

It's hard to ignore that in your paper Table 3, the psnr of Sony rgb input is 17.4 while another psnr is 25.11. I couldn't understand why they differ so much when both are rgb images. I mean，if input rgb images, they must have gone through the same process before put into the network.
And would you mind to explain how you convert your raw datasets to rgb images, and how to understand this difference?

why ignore negative values in pack_raw() function?

@cchen156 Hi, nice work and congratulations. I am wondering why you use np.maximum(xx,0) in pack_raw() function to ignore negative values. Will it affect the result?

The result of reimplementing the paper with pytorch is very bad?

I reimplemented the paper with pytorch,but the result is very bad.I am not sure whether pytorch has changed the effect of tensorflow in this paper.

Can we execute the repo over any cloud platform.

The quantization

Hi,

Thanks for your novel work. I have some questions. I noticed that you restrict the value of ground truth image around 0 to 1, then force the model to generate output image which is also around 0 to 1. I am confused that why not just use the original ground truth image as the label and produce the output image directly without multiplying 255. Is there any difference between these two ways? Could you please help answer this question?

Thanks,

About rgb training

paper has sRGB input comparision, I modify the last two level conv10(12 channel ) to output 3channel for the rgb input image generated by rawpy.
the training loss is down to 0.025, the result is not bad, but some image is not real. is this correct way?

Siny Test result : the ouptput image is a part of the gt_image

I have try to test the Sony Image Data:

/home/ljz/桌面/DB3065DB.png

Why output picture so dark!!! I use the pretrained model. Need any other operation ???

I download the pretrained model and run 'test_Sony.py'.But the output is very dark!

Could you provide training test data set within 2G.

Hello,thank you for your help.Because my computer configuration is too low, RAM is only 4G, gpu memory 2G, although you provide sonny(12G) data set, but still a bit big for me and I can't download it .so could you provide training test data set within 2G. I just want to use a small amount of data to test the model.
Thank you !