yuanming-hu/exposure

Learning infinite-resolution image processing with GAN and RL from unpaired image datasets, using a differentiable photo editing model.

I'm Yuanming Hu. Working on Taichi Lang :-)

---

A curated list of awesome Taichi applications, courses, demos and features: Awesome Taichi.

exposure's Issues

Undo gamma correction

Hi @yuanming-hu ,why do you do 'linearize_ProPhotoRGB' or 'Linearize sRGB' first for the input images?
I tried to change the paramater from 1.001 to 2.2, the results differ a lot. I wonder what's the reason of reference for this 'linearize'. Thanks a lot!

Pretrained model Miss

Hello @yuanming-hu
Thanks for your code! I'm studying relative exposure topic for stitching task.
The pretrained model in the models folder is missing. Do you have any trained models now?
I would really appreciate it if you can share one trained tensorflow model.
Thank you!

Did you forget to drop the critics.py/cnn out?

Did you forget to drop the critics.py/cnn out? I have not found it.
And Why it's the uniform random when select the action(filter), but not the normal random?
Thanks.

How to train the style learning experiments with my own data?

Plz give me some reference,thanks

if remove the rl part, just use a fixed pipeline, will the result be similiar

Hi, I'm wondering if remove the rl part and just use a fixed pipeline with wgan, will the fixed pipeline generate a similar result.

SaturationPlus Filter Parameter returns 0

On all of my results, I am seeing in the .Steps. image that the parameter used for the SaturationPlus filter is always 0.00. Sometimes it's 0.01, but it's never enough to make a meaningful difference in the next image.

I thought that maybe it was a problem with the data I used to train, but when I looked at the pretrained example's outputs, I saw the same issue with the SaturationPlus filter on those .steps. images.

That's the problem I'm experiencing right now. If anyone can offer some guidance, I would appreciate it. Now I'll describe some of the steps I've tried to take to fix it and explain why they haven't worked:

I noticed that Tensorflow now has a function for adjusting the saturation of images directly:

enhanced_s = tf.compat.v1.image.adjust_saturation(img, scale)

where scale is the multiplier applied to the input image's saturation. I tried replacing the process() function of the SaturationPlusFilter() class in filters.py with this function, of course there's no preexisting gradient for the adjust_saturation() function so I just hardcoded the scale value to 1.5 and used the output param to linearly interpolate between the input and enhanced images as Yuanming-Hu did, but the network still doesn't learn how to properly use the filter after 20000 iterations of training.

Quantitative evaluation using histogram_intersection.py

Hi @yuanming-hu ,
Thanks for sharing your codes. Could you please explain a little bit on how to obtain the quantitative evaluation results using histogram_intersection?
I tried to feed two directories as the input arguments and modified line 51 to the following:
patch = cv2.resize(new_image, dsize=(80, 80), interpolation=cv2.INTER_AREA),
but got an error at line 30 inside get_histograms(images). The error message was: TypeError: iteration over a 0-d array.
Thanks!

Further simplify & enhance code

• Fix #15
• Enable #13

Baoyuan Wang link is invalid

404 error

how to train on my own dataset?

What's the output node name for the saved checkpoint?

Hi,

What's the output node name for the checkpoint file? I am trying to freeze the model as a protobuf file.

Thanks

Question on how to train my own model

Hi,
When I am training my model, I meet a problem:

FileNotFoundError: [Errno 2] No such file or directory: '/data/yuanming/fivek_dataset/sup_batched80aug_daylight/image_raw.npy'

I am confused about what image_raw.npy, image.npy and image_retouched.npy means and how to build my
own image_raw.npy, image.npy and image_retouched.npy ?

Also, can anyone give me some guides on how to train my own model? Thanks so much!

Tonefilter部分理解问题

你好，我在阅读tonefilter部分代码时候有一点疑问，就按照原定的8阶tone_curve为例，将原始图像按照亮度分级为8级对应为Ri， 输出结果Rio = （Ri * lut[i] + sum（j=0-i）(lut[j] * 1 / 8）） * 8 / lut_sum, 感觉这样的公式有点奇怪，Ri-1中最大值和Ri中的最小值的输出应该是连续的更为合理？

之前算错了，是连续的

Critic gradient norm and penalty

Hi, I'm reading the code of GAN net, I cannot understand the meaning of "Critic gradient norm and penalty"part, do you have any reference? Thanks a lot!

license ?

Hi @yuanming-hu
I'm studing the relative task in stitching, keep continuous attention on your pro. what is your license category？Thanks.

RAW Camera file exports are very small

When I process from Nikon RAW files the exported files are in PNG but very small, in the order of 150x200 or something like that. Can I modify something to export bigger?

Thanks!! and great job here

Is there any chance to get this working in Win64?

I tried but I get an invalid syntax at util.py, line 658

Thank you in advance

I am confused about the reward function.

Discriminator discriminates the final result. How can it produce reward for each operation in the retouched processing?

关于训练

你好，很感谢你提供了代码。我根据提示的步骤进行训练，但是在训练到10390次的时候就卡住了，无法再接着训练，电脑也卡死，并且g_loss为负，请问你是否知道这是什么情况？谢谢~

image data type float64 is not supported. Please email Yuanming Hu.

Any other data that you need?

Also, can we down sample and if so, what formats are supported?

Steps for pre-processing the Five-K dataset.

FiveK dataset

Hi,
Thank you for releasing the code for this paper.
In addition to Artist C in the FiveK dataset, do you also have zip files available for the other 4 artists?

Thanks!

please help me......

D:\exposure-master\exposure-master>python train.py example test
Scripts are backed up. Initializing network...
Traceback (most recent call last):
File "train.py", line 18, in
main()
File "train.py", line 13, in main
net = GAN(cfg, restore=False)
File "D:\exposure-master\exposure-master\net.py", line 44, in init
self.memory = ReplayMemory(cfg, load=not restore)
File "D:\exposure-master\exposure-master\replay_memory.py", line 12, in init
self.real_dataset = cfg.real_data_provider()
File "D:\exposure-master\exposure-master\config_example.py", line 198, in
set_name='2k_target')
File "D:\exposure-master\exposure-master\artist.py", line 46, in init
idx = list(map(int, idx))
ValueError: invalid literal for int() with base 10: '# Note: this list is 1-based, i.e. ids are among [1, 5000]\n'
D:\Anaconda3\envs\tensorflow\lib\site-packages\h5py_init_.py:34: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters

Variable "features" in net.py

Hi, yuamning-hu
Thank you for openning implementation code.

I have a question

In the net.py line 325 - 354,

      for j in range(giters):
        feed_dict, features = self.get_training_feed_dict_and_states(iter)
        if iter == 0:
          feed_dict[self.lr_g] = 0
        feed_dict[self.progress] = progress
        _, g_loss, v_loss, fake_output, new_states = self.sess.run(
            [(self.opt_g, self.opt_v), self.g_loss, self.v_loss,
             self.fake_output, self.new_states],
            feed_dict=feed_dict,
            options=run_options,
            run_metadata=run_metadata)
        if self.cfg.supervised:
          ground_truth = feed_dict[self.ground_truth]
        else:
          ground_truth = None
        self.memory.replace_memory(
            self.memory.images_and_states_to_records(
                fake_output, new_states, features, ground_truth=ground_truth))
        v_loss_pool.append(v_loss)
        g_loss_pool.append(g_loss)

The variable named "features" seems always zeros. Is it meaningful variable ?

is it possible to replace the commercial preprocessing by an invokable API?

Hi, very appealing work, and I would like to know whether it is possible to replace the inconvenient commercial preprocessing (i.e. the operations implemented by Lightroom mentioned in your wiki by a invokable API, since in most cases we should start from the unprocessed raw data rather than the demosaicked one.

Virtually I tried to utilize the dcraw/libraw to preprocess the mosaicked raw data into the ProPhoto RGB space, but unfortunately no decent result is obtained by post-processing with the 'exposure' framework.

Besides, I was also wondering how you can ensure the Adobe LightRoom behaves as intended? In my sense, the aimed preprocessing consists of three steps as follow:

1. white balance 2) demosaick the (bayer) raw data 3) convert the linear camera-specific raw space into linear ProPhoto RGB space by a 3x3 color transformation matrix.

Nevertheless, it seems the additional step, \e.g. gamma correction is also applied by LightRoom, since I find in your implementation, the linearizing is necessary to invert the effect of non-linear gamma correction. I also have no idea how you can figure out the parameters of gamma correction there (is that just an approximation like 1/2.2?)

In summary, my question is what you exactly intended to do in Adobe Lightroom, and can it be replaced by a invokable API?

Thank you very much!

why for 'linearize'

Hi, @yuanming-hu, I wonder what's the reason or reference for doing 'linearize_ProPhotoRGB' or 'Linearize sRGB' first for input ? I changed the paramater from 1.001 to 2.2, the results differ a lot. Thanks a lot!

How to convert other imgs through the parameters we get from one img inferring?

Using opencv or photoshop? Parameters include exposure, contrast, color curve, tone curve, and gamma value. @yuanming-hu

Segmentation fault while runing evaluation code

Hi, Yuanming,

While I run the command at Using the pretrained model section , I get segmentation fault error message.

There is my platform and package information:

• ubuntu 18.04
• GPU: K80
• nvidia driver version 390.77
• python 3.6
• tensorflow-gpu 1.11.0
• opencv-pytohn 3.4.3.18
• tiffile 0.15.1
• scikit-image 0.14.1

The following is error message from dmesg:

"segfault at 0 ip 00007fce957b5216 sp 00007fccd9ff1480 error 4 in _pywrap_tensorflow_internal.so[7fce90acb000+2aa45000]"

And I found that seg fault might be occurred at Line 804(self.sess.run) in net.py.
Do you have any idea how to solve it?
Thanks for your help.

Improve export speed

input as 8bit jpg?

Have you tried 8bit jpg as input? If so, How about the performance？

Did the extra planes help?

Hi, have you compared the effect of inputs with and not with the extra planes (average luminance, contrast and saturation)? Did them really help?
Thank you.

how to train it in supervised version

When i change 'cfg.supervised' , it makes mistakes.
Traceback (most recent call last):
File "train.py", line 18, in
main()
File "train.py", line 14, in main
net.train()
File "/home/syk/exposure/net.py", line 373, in train
run_metadata=run_metadata)
File "/home/syk/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/home/syk/anaconda3/envs/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1096, in _run
% (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (64, 64, 3) for Tensor 'fake_input:0', which has shape '(?, 64, 64, 3)'

Question about value network

Hi Yuanming,

Thanks for releasing codes of this wonderful project!

I have a question about the value network. In net.py, the new_value is predicted by observing fake_output and new_states. Let s_t denote fake_input, and then fake_output is s_{t+1}. The new_states contain the ation a_t that transfers s_t to s_{t+1}. Therefore, it seems the codes are predicting Q(s_t, a_{t-1}), Q(s_{t+1}, a_t) rather than Q(s_t, a_t), Q(s_{t+1}, a_{t+1}). If so, I am confused how the policy gradients are calculated (e.g., Eqn. (7) in the paper). I might get something wrong. I'd appreciate it if you could help me clarify this question. Thanks!

Yu Ke

How can I do reverse-engineer?

@yuanming-hu

Testing procedure gets stuck at "initializing..."

I train a model with my own dataset and stored the checkpoint in ./models/sintel/test

Then I test the model using command " python3 evaluate.py sintel test path/to/mydata/*.JPG"
The testing procedure then gets stuck at "Initializing..." and takes up only 99MB of gpu.

What's the problem and how should I fix it? Thanks!

How to generate the training data from the very begining

Hi yuanming, I'm try to generate the training data from fivek origin data and I'm come into some problems, you said
In the Collections list, select collection Inputs/Input with Daylight WhiteBalance minus 1.5.
but I'm confused at minus 1.5, what does minus 1.5 mean? Minus tone? tint? or exposure?

Could you pls provide a pretrained model ?

As title. Really interest in your work. Many thanks.

the parameters of each filter

Hello, I don't quite understand the π2 in your paper. Does it mean that we need to learn the parameters of each filter??

数据集无法下载

Adobe5K数据集太大了下载不下来，然后在“Preparing data for the MIT Adobe FiveK Dataset with Lightroom”中提到的PartI和PartII数据集也无法下载下来，您能提供一个百度网盘的链接吗？

CNN function Approximate

In experiments, have you ever compared the effect of CNN with different layers in this task? 4 is the minimum requirements considering performance？Thanks!

Visualizing results for the 1000 RAW test images?

Hi, @yuanming-hu. Thanks for sharing the codes.
I'm wondering if there is an easy way to visualize the results for those 1000 RAW test images (part 3 of the MIT-Adobe FiveK Dataset).
Thanks.

Training the model, train vs. test data

Hello!

I have been looking at the config_sintel.py file a bit and am not understanding how the train/test data are separated. In your slides, you suggest that this model can be trained using exclusively the retouched image data, but the train.py function seems to expect matched pairs of images with the untouched data being in one folder, the retouched data in a second folder... What about the test data? In the config_sintel.py file, the fake_data_provider_test folder is the same as the training folder, while the comment above it specifically says not to do this.

Right now I am thinking I should have three folders in my model directory, like this:

cfg.fake_data_provider = lambda: FolderDataProvider(folder='models/ColorCorrect/Uncorrected', default_batch_size=cfg.batch_size)
cfg.fake_data_provider_test = lambda: FolderDataProvider(folder='models/ColorCorrect/Validation', default_batch_size=cfg.batch_size)
cfg.real_data_provider = lambda: FolderDataProvider(folder='models/ColorCorrect/Corrected', default_batch_size=cfg.batch_size)

Where the Uncorrected Folder has 300 untouched images, the Validation Folder has 50 different untouched images, and the Corrected Folder has a total of 350 images, all matching to either the contents of the Uncorrected or Test Folders.

Is this the correct approach?

Thanks for your time, and for sharing your very impressive work!

Some questions about mask

Hi, thank you for you sharing code! I found that exposureNet can generate a mask when generate a filter parameters, however, only 5-6 parameters and get_mask() function can produce a HW1 mask. There is no hint in paper about how to train the mask and how to generate the mask, so can you help me to explain theory about the mask using in exposureNet.

Thank you very much!

How can I train the model using my own data?

For example, I have two datasets of PNG images. How can I use the code to train the model, which will transfer the style of one dataset to the other one (unpaired)?

Thank you!

Differentiable hsv to rgb

Hi, thank you for your great work and for making the code available. I notice you use the Tensorflow hsv_to_rbg and rgb_to_hsv functions (class SaturationPlusFilter(Filter)) for adjusting the saturation of the images. However, function hsv_to_rgb does not appear differentiable due to the rounding function. How do you deal with this in your method?

How can I convert the input image from ProPhoto RGB color space to sRGB color space ?

Or, can I convert the test image from sRGB color space to ProPhoto RGB color space for inference ?

WGAN error

Any idea if this is related to the new version of tensorflow?

tensorflow 1.0.1
tensorflow-gpu 1.10.0

** WGAN
routine_loss (?, 1)
pg_loss (?, 1)
Traceback (most recent call last):
File "evaluate.py", line 35, in
evaluate()
File "evaluate.py", line 27, in evaluate
net = GAN(cfg, restore=True)
File "/home/licarazvan90/exposure/net.py", line 174, in init
alpha_dist = tf.contrib.distributions.Uniform(low=0., high=1.)
TypeError: init() got an unexpected keyword argument 'low'

How is your network initialized?

Hello, thanks for your opening codes.
I want to know how is your network initialized before training?
Thanks
Looking forward to your reply

About Luminance Calculate Formula

Hi, Yuanming

Regarding to your code, I have few questions towards the formula you are using for calculating luminance,

I noticed that you are using 0.27*R + 0.67*G + 0.06*B(I could only find reference of this formula by http://www.cs.utah.edu/~reinhard/cdrom/tonemap.pdf), however, I also read some other articles about luminance formula, seems 0.2126*R + 0.7152*G + 0.0722*B is more often used for linear RGB cases, and 0.299*R + 0.587*G + 0.114*B for gamma-corrected RGB cases.

could you give some explanation about your luminance formula? and why you are using the same formula for both cases in your code?

Evaluate an image with 3500ms-4000ms. Not 30ms?

I use the provided eval() function in net.py to evaluate some JPG images with sizes around 500x500.
The evaluation time of an image I calculated is on average about 3.5s-4s on my GPU. And each retouch process costs 700ms to 800ms.

However you say An unoptimized version takes 30ms for inference on an NVIDIA TITAN X (Maxwell) GPU.
I understand that you evaluate and get a retouched photo only takes 30ms? Is my understanding correct?
How can I speed up if you test it in 30ms?