cfernandezlab/cfl

Tensorflow implementation of our end-to-end model to recover 3D layouts. Also with equirectangular convolutions!

CFL: End-to-End Layout Recovery from 360 Images

Tensorflow implementation of our end-to-end model to recover 3D layouts of indoor scenes.

Introduction

This repository contains original implementation of the paper: 'Corners for Layout: End-to-End Layout Recovery from 360 Images' by Clara Fernandez-Labrador*, José M. Fácil*, Alejandro Perez-Yus, Cédric Demonceaux, Javier Civera and José J. Guerrero.

The page of the paper is https://cfernandezlab.github.io/CFL/

Citing

Please cite CFL in your publications if it helps your research:

@article{fernandez2020corners,
  title={Corners for layout: End-to-end layout recovery from 360 images},
  author={Fernandez-Labrador, Clara and Facil, Jose M and Perez-Yus, Alejandro and Demonceaux, C{\'e}dric and Civera, Javier and Guerrero, Josechu},
  journal={IEEE Robotics and Automation Letters},
  year={2020},
  publisher={IEEE}
}

News

Now implemented for TF2 on the TensorFlow addons package! Check the tutorial in this link!

New implementation in PyTorch as well here link!

Use Instructions

We recommend the use of a virtual enviroment for the use of this project. (e.g. pew)

$ pew new venvname -p python3 # replace venvname with your prefered name (it also works with python 2.7)

Install Requirements

1. This code has been compiled and tested using:

• python3
• g++-4.9
• cuda-10.0
• cuDNN 7.5
• TensorFlow 1.13

You are free to try different configurations. But we do not ensure it had been tested.

2. Install python requirements:

(venvname)$ pip install -r requirements.txt

3. Compile Deformable Convolutions:

We use this Deformable Convolutions implementation to create EquiConvs, a novel implementation of the convolution for 360º images that adapts the shape of the kernel accordingly to the equirectangular projection distortions. Valid for any task involving panoramic images!

First, make sure to modify Models/make.sh with your computer settings (i.e. CUDA_HOME, CUDA Architecture). Second run the following commands.

(venvname)$ cd Models
(venvname)$ bash make.sh

Download Dataset

SUN360: download

Download Models

StdConvs version: download

EquiConvs version: download

Run Test

Edit the config file so that the path to the dataset folder and the network parameters is correct.

To run our demo please run:

(venvname)$ python test_CFL.py --dataset Datasets/SUN360/test --weights Weights/StdConvs/model.ckpt --network StdConvs --results results_std # Runs the test examples with Standard Convolutions and save results in 'results_std'

(venvname)$ python test_CFL.py --dataset Datasets/SUN360/test --weights Weights/EquiConvs/model.ckpt --network EquiConvs --results results_equi # Runs the test examples with Equirectangular Convolutions and save results in 'results_equi'

From Corner Maps to 3D Layout

Check this notebook 2Dto3D.ipynb.

Contact

You can find our contact information in our personal websites Clara and Jose M.

License

This software is under GNU General Public License Version 3 (GPLv3), please see GNU License

For commercial purposes, please contact the authors.

Disclaimer

This site and the code provided here are under active development. Even though we try to only release working high quality code, this version might still contain some issues. Please use it with caution.

cfl's Issues

test deform conv fails

Hello,

Let me start by thanking you for the wonderful work and for making the code publicly available! Interested by your work, I have been trying to set up the pipeline. As mentioned in the readme, I installed all the dependencies in python and compiled the deform_conv_layer. But when I try to run the test_deform_conv.py, it fails with the following error:

tensorflow.python.framework.errors_impl.NotFoundError: ~/CFLE2E/CFL/Models/deform_conv_layer/deform_conv.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringB5cxx11Ev

From my previous experience, I had recognized this as being the result of a mismatched gcc that was used to compile tensorflow and the target custom layer. I installed tensorflow from the conda repositories. So I don't know which exact gcc was used to compile the binaries. I am using Arch, which comes with gcc 8.3. I am also using Cuda 10. I tried compiling the custom layer with both 4.9 (as mentioned) as well as gcc 5.5. However, both of them results in the same error. So it would be a great help if you could point me to a solution.

Hi. where can I get the model.ckpt ?

How to understand the use of K and inv_K?

Hello! Thanks for your great work.
When I read your code, I have some confusion.
I have seen the formula(3) and (5) in 4.1 EquiConvsDetails, and you define a matrix K = np.array([ [focal, 0, c_x], [0, focal, c_y], [0,0,1] ]) and rays = np.matmul(inv_K, rays.reshape(3, r_h*r_w)). I guess your aim is to put the coordinates on the sphere? But I don't understand the geometric principle behind. Can you introduce it to me? Thank you very much!

Question about pij in formula （3）

I am so sorry to trouble you again. I am not understand the code below.
According to the paper,I know rays is pij = [i j d]，rays[0] is corresponding to i, rays[1] is corresponding to j, … but I can't know the role of K and c_x, c_y. As you say in the paper, i and j are integer in the range [- (r-1)/2, (r-1)/2]. So why do you need to multiply inv_K and rays.
Another, the value of c_x and c_y is always 0?
c_x = 0 c_y = 0 K = np.array([[d,0,c_x],[0,d,c_y],[0,0,1]]) inv_K = np.linalg.inv(K) rays = np.matmul(inv_K, rays.reshape(3,r_h*r_w))

About results of EquiConvs and StdConvs

From the results of your paper, I find that the network constructed by Equirectangular Convolutions (EquiConvs) has no great improvement in indicators compared with the network constructed by traditional convolution (StdConvs). How can you explain the benefits of EquiConvs in processing ERP projection images?

In addition, the computation time of the network constructed by EquiConvs is 10 times that of the network constructed by StdConvs. So what are the advantages of EquiConvs compared to StdConvs?

help

Is there any training code based TF?

How to use optimizer in Equiconv?

Hello, Thanks for your great work!
In your code, I can't find train.py. My question is how to use optimizer. Because EquiConv is an op that we defined, so how to update it like we can directly use Adam to update tf.nn.conv2D

Questions about formula(5) in 4.1 EquiConvs

Thanks for your great work!
First, I meet some diffculty when I saw formula(5). I can't understand this formula, your aim is project each point in kernel into the sphere surface, but can you explain why your rotation matrix like this? And the code of rotation matrix is bot same with Rodrigues' Rotation Formula, I found when theta = pi/2, the result is different.
Second, have you trid DCN-V1 or V2?

A question about CE

Hi！
Thanks for your great work! I have a question about CE, hoping to get your help.
In your code, I can't find how to calculate CE and PE. The output of network is an image, but I wonder how to get the corner id from Corner Map, especially for complete segmentation, can you spare your code or describe your method. Thank you!
Best wishes for you.

How to recovery layout from the corner map and the edge map?

Question about formula （5） in 4.1 EquiConvs Details

Thanks for your great work!
First, I meet some diffculty when I saw formula(5). I can't understand this formula, your aim is project each point in kernel into the sphere surface, but can you explain why your rotation matrix like this? And the code of rotation matrix is bot same with Rodrigues' Rotation Formula, I found when theta = pi/2, the result is different.
Second, have you trid DCN-V1 or V2?

Mac air 2

Any recommendation how to launch this with mac?

I suppose the make.sh script is a CUDA supporting one only ?

Thanks for providing the code! Can you provide the training script please?

layout 3d

Hello, thanks for the code!

a small doubt, I followed the 2Dto3D python notebook and understood how to get the wireframe. can you please tell me what to change to get the room layouts in 3D like the teaser gif you have in this repo?

Thanks

Error in deformable_im2col？

Hi!
When I use EquiConv to train, an error occur: error in deformable_im2col: out of memory, and err = 2, cudaSuccess = 0, have you ever encountered this kind of error. Online solutions are ：change to a small model 、small dataset. If you are free
（1）Can you tell me the specific machine configuration during your training？ GPU……
（2） In your paper, the input of the network is image of 256128. So do you encountered similar errors when using larger images？If not, why image size is designed to be 256128?
（3）I know that EquiConvs are built on top of deformable convolutions, the only difference is the way to get offset. My question is whether we need to change the deformed convolution other than the way to find the offset？ I got the Deformablle_Convolution_pytorch from https://github.com/open-mmlab/mmdetection, then I only to change the way to find offset in x.py? And no need to change .cu and .cpp files that they define this op(DCN).
（4）Can you share train.py with me? I want to use your code to test whether equiconv can run on my computer. It ’s okay if it ’s inconvenient.
Thank you very much for your help during this time.