liagm/lfattnet

Attention-based View Selection Networks for Light-field Disparity Estimation

LFattNet: Attention-based View Selection Networks for Light-field Disparity Estimation

Attention-based View Selection Networks for Light-field Disparity Estimation

Yu-Ju Tsai,¹ Yu-Lun Liu,^1,2 Ming Ouhyoung,¹ Yung-Yu Chuang^1
1National Taiwan University, ²MediaTek

AAAI Conference on Artificial Intelligence (AAAI), Feb 2020

• (Paper Link)
• Contact: [email protected], [email protected]

Network Architecture

SOTA on 4D Light Field Benchmark

• We achieve TOP rank performance for most of the error matrices on the benchmark.

• For more detail comparison, please use the link below.
• Benchmark link

Environment

Ubuntu            16.04
Python            3.5.2
Tensorflow-gpu    1.10
CUDA              9.0.176
Cudnn             7.1.4

Train LFattNet

• Download HCI Light field dataset from http://hci-lightfield.iwr.uni-heidelberg.de/.
• Unzip the LF dataset and move 'additional/, training/, test/, stratified/ ' into the 'hci_dataset/'.
• Check the code in 'LFattNet_func/func_model_81.py' and use the code at line 247.
• Run python LFattNet_train.py
  • Checkpoint files will be saved in 'LFattNet_checkpoints/LFattNet_ckp/iterXXXX_valmseXXXX_bpXXX.hdf5'.
  • Training process will be saved in
    • 'LFattNet_output/LFattNet/train_iterXXXXX.jpg'
    • 'LFattNet_output/LFattNet/val_iterXXXXX.jpg'.

Evaluate LFattNet

• Check the code in 'LFattNet_func/func_model_81.py' and use the code at line 250.
• Run python LFattNet_evaluation.py
  • To use your own model, you can modify the import model at line 78 like below:
    • path_weight='./pretrain_model_9x9.hdf5'

Citation

@inproceedings{Tsai:2020:ABV,
        author = {Tsai, Yu-Ju and Liu, Yu-Lun and Ouhyoung, Ming and Chuang, Yung-Yu},
        title = {Attention-based View Selection Networks for Light-field Disparity Estimation},
        booktitle = {Proceedings of the 34th Conference on Artificial Intelligence (AAAI)},
        year = {2020}
}

Last modified data: 2020/09/14.

The code is modified and heavily borrowed from EPINET: https://github.com/chshin10/epinet

lfattnet's Issues

What is the GPU implementation in the evaluation?

Hi, the LFattNet is interseting. However, it is reported out of memory when I test the network using the 512x512 HCI light field data (with a 2080ti GPU). I am wandering that how to use a single 1080ti GPU to test the network as the your paper, thanks!

Questions about training

Dear LIAGM,
I have some questions about the training code：

1. In the code, the epoch is set to 10000000. How long does each epoch takes, and how many epochs need to be trained in total?
2. I fit the environment as your describe, but an OOM error occurred. My GPU memory is 11G. Finally, I adjusted the number of 3D convolution kernels from 150 to 100 to run successfully.
   Looking forward to your reply.
   Thanks.

Asking a question about the LFattNet

Out of Memory? what's the problem?

Dear Liang,
My test PC is like this: one NVIDIA GTX 1080Ti GPU, CPU: intel i5, Memory: 8GB
but when I run it , it show : GPU out of memory.
Regards,

How to resolve this?

num_samples = int(nest.flatten(data)[0].shape[0])
AttributeError: 'threadsafe_iter' object has no attribute 'shape'

pytorch code

hello,do you have pytorch code?

A question about the LFattnet?

How to modify？

ValueError: Error when checking model target :the list of numpy arrays that you are passing to your model is not the size the model expected.

Regarding LFattNet_sub

Hello,

I notice that an algorithm called LFattNet_sub was currently submitted to the HCI 4D LF benchmark and achieved top performance. I wonder how the method is modified to achieve such a significant improvement?

How to imlement operation?

Dear LIAGM,
Can the code below implement the operation? is it right ? I use this in the program, but find the Cost volume became Nan !

Thanks

Open Cv

hell , when i installed the required libraries i found this error related to OpenCV even if i actually installed the open cv
import cv2
Import Error: DLL load failed: The specified module could not be found.

so can you say the versions of the libraries you used and if you know the solution for this error please tell me?

Or the Best solution is to provide us with the docker image of your environment so that we can work correctly because the TensorFlow 1.10 is very old and we face a lot of troubles when installing it and it produces a lot of dependencies and compatibility errors.

thanks a lot.

Asking for help about the trainning result?

作者你好，我在尝试重新运行您的代码时候，因为我下载不到tf1.10版本，需要1.13

Can you share the checkpoint in the real-world dataset？