marcelampc/d3net_depth_estimation

Dense Deep Depth Estimation Network (D3-Net) in PyTorch.

Deep Depth-from-Defocus (Deep-DFD)

Network Architecture

This code implements the Dense Deep Depth Estimation Network (D3-Net) in PyTorch, from the paper:

On regression losses for deep depth estimation, Marcela Carvalho, Bertrand Le Saux, Pauline Trouvé-Peloux, Andrés Almansa, Frédéric Champagnat, ICIP 2018.

Fig.1 - D3-Net architecture.

If you use this work for your projects, please take the time to cite our ICIP paper:

@article{Carvalho2018icip,
  title={On regression losses for deep depth estimation},
  author={Marcela Carvalho and Bertrand {Le Saux} and Pauline Trouv\'{e}-Peloux and Andr\'{e}s Almansa and Fr\'{e}d\'{e}ric Champagnat},
  journal={ICIP},
  year={2018},
  publisher={IEEE}
}

Indoor and outdoor DFD dataset

We also publish the dataset for Deep Depth from Defocus estimation created using a DSLR camera and a Xtion sensor (figure 1). This dataset was presented in in:

Deep Depth from Defocus: how can defocus blur improve 3D estimation using dense neural networks?, Marcela Carvalho, Bertrand Le Saux, Pauline Trouvé-Peloux, Andrés Almansa, Frédéric Champagnat, 3DRW ECCV Workshop 2018.

The dfd_indoor dataset contains 110 images for training and 29 images for testing. The dfd_outdoor dataset contains 34 images for tests; no ground truth was given for this dataset, as the depth sensor only works on indoor scenes.

Fig.2 - Platform to acquire defocused images and corresponding depth maps.

---

BibTex reference:

Generate Synthetic Defocused Data

In generate_blurred_dataset.m, change lines 14 to 18 to corresponding paths in your computer and run.

If you use this work for your projects, please take the time to cite our ECCV Workshop paper:

@article{Carvalho2018eccv3drw,
  title={Deep Depth from Defocus: how can defocus blur improve {3D} estimation using dense neural networks?},
  author={Marcela Carvalho and Bertrand {Le Saux} and Pauline Trouv\'{e}-Peloux and Andr\'{e}s Almansa and Fr\'{e}d\'{e}ric Champagnat},
  journal={3DRW ECCV Workshop},
  year={2018},
  publisher={IEEE}
}

---

Requirements

Requires Python 3.6 with pip and the following libraries:

• Linux
• Python 3.6+
• PyTorch 1.9
• Cuda 10.2
• Visdom

Depth Estimation

Installation

Using Conda:

cd pytorch
conda create -n d3net-env python=3.7 -y
conda activate d3net-env
conda config --append channels conda-forge
sh pre_install.sh

Create a checkpoints and results folder, or a symbolic link to your respective folders.

mkdir checkpoints results

# Or
ln -s path_to_checkpoints checkpoints
ln -s path_to_results results

Train D3-Net on DFD Dataset

Run the script in train_dfd.sh to train D3-Net on DFD Indoor Dataset. In your terminal, enter your conda environment

# Start Visdom
visdom -p 8001

# In another terminal, if not already inside,
cd pytorch

# Run train
sh std_scripts/train_dfd.sh

Usage

Generate Synthetic Defocused Data

In generate_blurred_dataset.m, change lines 14 to 18 to corresponding paths in your computer and run.

[To be added]

License

Code (scripts and Jupyter notebooks) are released under the GPLv3 license for non-commercial and research purposes only. For commercial purposes, please contact the authors.

d3net_depth_estimation's Issues

matlab code

In the refoc_image.m, the line 70 Mk=Mk.*(1-conv2(Ad(:,:,k-1),PSF{k-1},'same')), but in your paper, the Mk is from the k + 1 layer to K layer. I wonder why it is different from your paper.

visualizing images during training

Hi thanks for the great work.
I noticed that the "[visualizer.py]" file provides image visualizing during training and evaluation, can you provide an example how to use this visualizing function?

Thanks

loss function

I want to train the model to predict blur image depth, so I wonder the loss function when you train your model ? thank you!

ask for help

How to use the code generate_blurred_dataset.m?

the interval of inputs and ground truth

My depth ground truth are true depth values (from 0m to 10m), but the outputs of tanh are within [-1, 1], I notice your dataset is also NYU v2. So I want to ask what is your depth ground truth? By the way, are your input images in the interval [0, 1] or [0, 255]? I am looking forward for your answers. Thank you!

What's the parameter.m for your paper?

Hi,

What's the parameter.m for you paper? I can't correspond your variable name with the camera parameters. Would you please give parameters used in paper for NYU? Thank you!

about synthetic dataset

hello, thanks for your work, I found the code you provided cannot produce the correct synthetic dataset in the paper, the pixels are blurred everywhere, is there anything wrong?

What's the meaning of min_mean in generate_blurred_dataset.m?

d3net_depth_estimation/matlab/generate_blurred_dataset.m

Line 34 in 413dca7

if mean2(im) <= min_mean

Should line 34 be mean2(im) > min_mean?

running the network with pytorch

hi, thanks for the work!
I was trying to run the network with pytorch with "python main.py --dataroot [root] --name [testdata]", but failed to get any progress.
Do training datasets need to be downloaded and added to the file manually? Is it convenient to have any pretained network parameters open?

Test issues

Hi, thanks for the great work.
I've trained your network with my dataset, and wish to test its accuracy, however the following error occured.

But I failed to find "last_common_layer.0" name issues in the original code. Hope you can give some help.

Learned weights

Hello!

First of all, thanks a lot for your very nice work! Do you plan on releasing the trained weights of the network? So that I can test it and obtain results out-of-the-box, without the need of training it myself (I don't have that much GPU power).

Thanks in advance and kind regards!
Lorenzo

Test file error

Run the train_nyu_d file to finish the training and then test_nyu_d.train_nyu_d.After completing the training by running the sh file, the test_nyu_d file was executed to check the result, but an error similar to the attached image occurred. I don't think the model structure fits, can you help me?

depth map style

Hello, thanks for your work, which style for depth you are used in your paper?
style1 : 0 for near and 255 for far
style2: 0 for far and 255 for near

is it 1 or 2 ??? and it is the same style in your MATLAB code??