djl11/pwcnettensorflow

PWC Net Tensorflow: Tensorflow implementation of Pyramid, Warping and Cost Volume Networks based on CVRP 2018 paper

PWC Net TensorFlow

Tensorflow implementation of Pyramid, Warping and Cost Volume (PWC) Networks based on the paper presented at CVPR 2018.
Currently, main.py simply downloads the FlyingChairs Dataset and starts training, following the outlined schedule.
This code could easily be adapted to train on other datasets though.

Tested Environment

Ubuntu 16.04
Python3
Tensorflow 1.8
Cuda 9.0

Acknowledgements

This repo uses 3 custom written tensorfow ops in c++

The correlation op was taken from this tensorflow implementation of UnFlow by Simon Meister
The ppm and flo decoding ops were taken from this collection of tf ops, from the Computer Vision Group, Albert-Ludwigs-Universität Freiburg

Usage

python3 main.py

A tensorboard session will automatically be started in a new tmux window (so that the visualisations are still available after the python session has ended).
This tensorboard session will log the training/validation losses, as well as giffs of the flow as it trains.

Some general hyperparameters regarding the logging of data can be changed through task.py
Other hyperparameters relating to the training schedule can be changed in the constructor of network.py
The default training/validation split is to have 90% training, with 10% left for validation.

Example visualisations following training

From left to right, the images below indicate rgb image, ground truth flow, predicted flow, flow error

Examples from the training set:

Examples from the validation set:

Example Training Loss

This is an example of the loss when training on the full flying chairs dataset (no validation was used on this occassion).

pwcnettensorflow's Issues

Possible bug in optical flow estimator network

Hi @djl11,
Thanks for sharing your code.

The optical flow estimator network is a feed-forward network formed by stacking layers one over the other:

In the flow_estimator_forward_pass function, where the estimator network is defined, it says:

conv4 = tf.layers.conv2d(conv3,64,[3,3],(1,1),'same','channels_first',(1,1),tf.nn.leaky_relu)
ft = tf.layers.conv2d(conv4,32,[3,3],(1,1),'same','channels_first',(1,1),tf.nn.leaky_relu)
wt = tf.layers.conv2d(conv4,2,[3,3],(1,1),'same','channels_first',(1,1),tf.nn.leaky_relu)

That is, both ft and wt are stacked over conv4, instead of being stacked one over the other as in the figure from the paper. Was this intentional, or a bug?

What's your final EPE error on FlyingChairs?

Trained weight

Thank you for making TensorFlow Version of PWC_Net.
Can you share your trained weight?
Also got the error when running main.py

fine-tune on the Kitti dataset

hi @djl11
Thanks for your work. I want to know if I want to fine-tune on the Kitti Dataset, the ground truth flow are sparse. so how can I change the Loss Function.