elliottwu/deephdr

This is the implementation for Deep High Dynamic Range Imaging with Large Foreground Motions (ECCV'18)

deephdr's Issues

How to compute HDR-VDP-2?

HDR-VDP-2 is used to measure perceived visual quality of the HDR image in the actual luminance domain (with the unit of cd/m^2) instead of the normalized domain. In this work, since the luminance maps are normalized, how to compute HDR-VDP-2?
matlab HDR-VDP-2:
function res = hdrvdp( test, reference, color_encoding, pixels_per_degree, options ).
Which color_encoding did you choose?
Thanks.

difference in display of hdr image

I run test.sh with your pretrained model, but got a different result with yours. The following results are my test result (upside) and yours (downside):

So what is the tonemap function that you use to display the final hdr image ?

static LDR image in training set

Hi, thanks for providing the way of downloading Kalantari's dataset.

I find that there are three static LDR images named ref_i.tif in each training scene. While I also down dataset from Kalantari's project web, and static LDR images are not provided.

So, are these pictures the original static images for generating the ground truth HDR? Or are them produced from ground truth HDR by apply inverse gamma correction?

why the size of output becomes small when test

1 when i set the --test_h 682 --test_w 1024
ValueError: Dimension 1 in both shapes must be equal, but are 170 and 171 for 'generator/decoder/concat_1' (op: 'ConcatV2') with input shapes: [1,170,256,128], [1,171,256,128]

COOL AND EASY, but how to SPEED UP?

First, thank you for your great contribution to the HDR.

Your code is tremendously cool and very clear so it is easy to use even like me.

Comparing with other HDR codes based on deep learning, I think distinctive feature of yours is you use 3 inputs with different exposure so output image can easily reconstitute almost-white and almost-black area of input images.

However, I wonder how to speed up the process.

With 3 800x600 jpg inputs, it takes 0.39s per one output. (i7-7700 3.60Hz, GTX1080 8G)
(Only 5% of my GPU is used)

I want at least 5 FPS

For poor newbie, please give me some advice.

OutOfRangeError

Hello,
Have you encountered outofrangeerror?
OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 8, current size 3) [[node shuffle_batch (defined at /mnt/Data/wk_linux/DeepHDR_T/DeepHDR/load_data.py:85) ]]
file "/mnt/Data/wk_linux/DeepHDR_T/DeepHDR/model.py", line 120, in build_model self.in_LDRs, self.in_HDRs, self.ref_LDRs, self.ref_HDR, _, _ = load_data(filename_queue, config)
My system is ubuntu 16.04, tensorflow 1.13.1, python 3.5
I used my own dataset which has higher resolution, it contains 80 sets of images. The images are .png format.
Looking forward to hearing some solutions from you, thanks.

i have two question about the code

Thank you for your work.
I have two question about the code.
1.In your paper, you have trained data by GPU.But when I run the code which you provide,the GPU is not work.
2.I don`t know how to align my pictures use homography transformation,can you provide to me your code?

difference of PSNR

Hi, thanks for your work.
I have a problem about the PNSR in your work.
the code about psnr is following one in mode.py

            curr_psnr = [compute_psnr(tonemap_np(res_samples[i]), tonemap_np(batch_ref_HDR[i])) for i in range(batchSz)]

I understand it is calculation of PNSR_T. While I tested the pretrained model in Kalantari's dataset and got a average PSNR_T (41.9977) which is different of the PSNR_T in your paper (40.81 for unet ver. and 41.65 for resnet ver.). I wonder if this is normal or the code of calculate PSNR_T is something other?

The detail of my testing in following:

First I cloned the projection and downloaded the pretrained model and Kalantari's dataset (from the link in readme.md)
Then I have the test dataset and pretrained model

$ ls  dataset/test
001  002  003  004  005  006  007  008  009  010  BarbequeDay  LadySitting  ManStanding  PeopleStanding  PeopleTalking
$ ls pretrained_ckp/
checkpoint  pretrained.data-00000-of-00001  pretrained.index  pretrained.meta

I made a test by the command

$ CUDA_VISIBLE_DEVICES=3 python test.py --dataset ./dataset/test --num_shots 3 --checkpoint_dir pretrained_ckp --batch_size 1 --results_dir results_pretrained --test_h 960 --test_w 1440

the log is following:

$ CUDA_VISIBLE_DEVICES=3 python test.py --dataset ./dataset/test --num_shots 3 --checkpoint_dir pretrained_ckp --batch_size 1 --res
ults_dir results_pretrained --test_h 960 --test_w 1440
2019-11-30 14:11:41.696733: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2019-11-30 14:11:43.051910: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: 0000:db:00.0
totalMemory: 15.78GiB freeMemory: 15.36GiB
2019-11-30 14:11:43.051957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2019-11-30 14:11:43.606928: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-30 14:11:43.606977: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0
2019-11-30 14:11:43.606986: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N
2019-11-30 14:11:43.607332: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14877 MB memory) -> physical GPU (device: 0, name: Tesla V100-PCIE-16GB, pci bus id: 0000:db:00.0, compute capability: 7.0)
WARNING:tensorflow:From /data/nonu_data/pyvenv/.venv/lib/python3.5/site-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:198: retry (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Use the retry module or similar alternatives.
WARNING:tensorflow:From /data/nonu_data/pyvenv/.venv/lib/python3.5/site-packages/tensorflow/python/util/tf_should_use.py:118: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
 [*] Reading checkpoints...
batch no. 1:
time: 4.8884
PSNR: 48.6350

batch no. 2:
time: 0.7845
PSNR: 44.5190

batch no. 3:
time: 0.2106
PSNR: 41.9392

batch no. 4:
time: 0.2477
PSNR: 42.9087

batch no. 5:
time: 0.2121
PSNR: 43.0685

batch no. 6:
time: 0.2460
PSNR: 38.9594

batch no. 7:
time: 0.2231
PSNR: 39.3789

batch no. 8:
time: 0.2155
PSNR: 34.9346

batch no. 9:
time: 0.2113
PSNR: 42.4318

batch no. 10:
time: 0.2115
PSNR: 37.4935

batch no. 11:
time: 0.2123
PSNR: 39.1859

batch no. 12:
time: 0.2146
PSNR: 41.9842

batch no. 13:
time: 0.2251
PSNR: 49.1725

batch no. 14:
time: 0.2523
PSNR: 43.7752

batch no. 15:
time: 0.2158
PSNR: 41.5797

Average PSNR: 41.9977

Data preparation

Hi,

Thank you for your nice work.
I found that in the paper the data is normalized to 0-1 but in your source code the range is from -1->1.

Can you give me an idea about this difference?

Thanks!

Homography align

Hi,
I a Masters student researching at a Signals Processing lab. In your paper, you mention homography alignment used prior in data preparation. However I could not find the corresponding code for this data preprocessing part. Could you please provide me with the code if it is available?

Thank you!

about your tf.pad

Thank you for your work. in function conv2d, why do you calculate pad_top, pad_bottom, pad_left, pad_right. why not use d_h=2, d_w=2 directly????

Failed to download dataset

Hello! It seems that the link to download the dataset is deprecated. Is that possible to provide another link?

raw

Hi,
Thanks for your job. I have some questions . The paper describes that the radiometric calibration and gamma correction are needed if the LDR is not in RAW format, however I don't see them in the code. Is the input of the network is in rgb format? Otherwise, If the input image is raw format, how to use the network? is demosaicking needed to convert RAW to RGB?

How did you generate the ground-truth for your test image

Hi,
I just have one question.
When I am trying to test the network with some random captures, I don't know how to create the correct ground-truth for 3 static images I captured.
I simply used the triangle method in Kalatari's work, but my output is probably different from yours.
If I just read ref_hdr_aligned.hdr and convert it to png using gamma = 2.2, the output image looks very dark.
Can you share with me how you created the ground-truth for the test data you captured?
Thanks a lot.

How to transfer a LDR image to HDR domain

I am confuse about how to transfer a LDR image to HDR domain.

In your paper, it was finished by gamma correction, (consistent with Kalantari's paper)
Hi = (Ii ** γ)/ti; γ=2.2, where ti is the exposure time of image Ii.

In your code,

def LDR2HDR(img, expo): # input/output -1~1
    return (((img+1)/2.)**GAMMA / expo) *2.-1

and

in_exps_path = os.path.join(scene_dir, 'input_exp.txt')
in_exps = np.array(open(in_exps_path).read().split('\n')[:ns]).astype(np.float32)
in_exps -= in_exps.min()
...
in_HDRs[:,:,c*i:c*(i+1)] = LDR2HDR(img, 2.**in_exps[i])

And the Kalantari's dataset include exposure bias {−2.0, 0.0, +2.0} or {−3.0, 0.0, +3.0}.

My confusion are follow:

1. Dose in_exps -= in_exps.min() mean that ti in formulation is the the exposure time relative to low exposure image ?

2. What is the exposure bias? Why exposure time is equal to 2**exposure_bias ?
   I search the wiki and find a formulation which may be related,
   EV = log2( N**2 / t )
   where EV is exposure value, N is f-number and t is exposure time. It can transfer to t = (N**2)/(2**EV). While it also can not explain the relationship between exposure time and exposure_bias.
   Dose the exposure_bias mean the exposure compensation ? If dose, then expousre_time = 2**exposure_compensation ?