kmaninis/osvos-caffe

One-Shot Video Object Segmentation

One-Shot Video Object Segmentation (OSVOS)

Visit our project page for accessing the paper, and the pre-computed results.

This is the implementation of our work One-Shot Video Object Segmentation (OSVOS), for semi-supervised video object segmentation. OSVOS is based on a fully convolutional neural network architecture that is able to successively transfer generic semantic information, learned on ImageNet, to the task of foreground segmentation, and finally to learning the appearance of a single annotated object of the test sequence (hence one-shot). Experiments on DAVIS 2016 show that OSVOS is faster than currently available techniques and improves the state of the art by a significant margin (79.8% vs 68.0%).

While the results of the paper were obtained by this code, we also provide a TensorFlow implementation of OSVOS: OSVOS-TensorFlow.

NEW: PyTorch implementation also available: OSVOS-PyTorch!

Installation:

1. Clone the OSVOS-caffe repository

   git clone https://github.com/kmaninis/OSVOS-caffe.git

   Skip the following line if you are using ubuntu 14.04! Compatibility with Ubuntu 16.04: If (and only if) you wish to run the code with Ubuntu 16.04, please run ./ubuntu_16.sh inside caffe-osvos.

2. Install the Caffe version under caffe-osvos/ along with standard dependencies, pycaffe and matcaffe. Caffe would need to be built with support for Python layers, in case you would like to use the Python API (TODO). cuDNN is not necessary.

   # In your Makefile.config, make sure to have this line uncommented
   WITH_PYTHON_LAYER := 1
   

3. Download the parent model from here (55 MB) and put it under models/.

4. Optionally download the contour model for contour snapping from here (55 MB) and put it under models/.

5. If you want to use the contour snapping step (a.k.a you downloaded the model of step 4.), run build.m from within MATLAB.

6. All the steps to re-train OSVOS are provided in this repository. In case you would like to test with the pre-trained models, you can download them from here (1GB) and put it under models/.

Demo online training and testing

1. Edit in file set_params.m the parameters of the code (eg. useGPU, gpu_id, etc.).

2. Run demo.m.

3. You can test all sequences of DAVIS 2016 validation set, by running test_all.m, once the pre-trained models are available under models/.

It is possible to work with all sequences of DAVIS 2016 just by creating a soft link (ln -s /path/to/DAVIS/) in the root folder of the project.

Training the parent network (optional)

1. All necessary files are under src/parent. So, cd src/parent.

2. Download the pre-trained vgg model by running ./download_pretrained_vgg.sh

3. Augment the data. In the paper we used flipping and scaling into 0.5, 0.8 and 1.0 of the original scale. Your image and ground truth pairs are specified in solvers/train_pair.txt.

4. Under solvers edit the data_root_dir of train_val*.prototxt.

5. Finally, train the parent model with python solve_cluster.py. You need pycaffe for this step, so don't forget to make pycaffe when installing Caffe.

Enjoy! :)

Citation

If you use this code, please consider citing the following paper:

@Inproceedings{Cae+17,
  Title          = {One-Shot Video Object Segmentation},
  Author         = {S. Caelles and K.K. Maninis and J. Pont-Tuset and L. Leal-Taix\'e and D. Cremers and L. {Van Gool}},
  Booktitle      = {Computer Vision and Pattern Recognition (CVPR)},
  Year           = {2017}
}

If you encounter any problems with the code, want to report bugs, etc. please contact me at kmaninis[at]vision[dot]ee[dot]ethz[dot]ch.

osvos-caffe's Issues

how to modify the caffe path

hi， can I install caffe by docker？And how to modify the caffe path in set_params.m?

What are the GPU memory requirements to run this program?

What are the GPU memory requirements to run this program?

Can this code run on a Nvidia TX1 with 4GB of memory?

What does 'DSN' mean (in train_val_stepX.prototxt)?

As title.
For example here

Unknown layer type: SigmoidCrossEntropyBalanceLoss

When I set osvos_params.useTrainOnline = 0 (use the pre-trained models), it works fine. But,
when i set osvos_params.useTrainOnline = 1, there is an error like this

Check failed: registry.count(type) == 1 (0 vs. 1) Unknown layer type: SigmoidCrossEntropyBalanceLoss (known types: AbsVal, Accuracy, ArgMax, BNLL, BatchNorm, BatchReindex, Bias, Concat, ContrastiveLoss, Convolution, Crop, Data, Deconvolution, Dropout, DummyData, ELU, Eltwise, Embed, EuclideanLoss, Exp, Filter, Flatten, HDF5Data, HDF5Output, HingeLoss, Im2col, ImageData, InfogainLoss, InnerProduct, Input, LRN, LSTM, LSTMUnit, Log, MVN, MemoryData, MultinomialLogisticLoss, PReLU, Parameter, Pooling, Power, Python, RNN, ReLU, Reduction, Reshape, SPP, Scale, Sigmoid, SigmoidCrossEntropyLoss, Silence, Slice, Softmax, SoftmaxWithLoss, Split, TanH, Threshold, Tile, WindowData)
*** Check failure stack trace: ***
Killed

use boundary snap only

Hi，thx for your pro. I‘m doing post processing after segmentation，but I do not have good ideas. You mention boundary snap in the paper，but I do not know how to use it to test the effect. Could you help me？

Augmented dataset

Is it convenient for you to upload your augmented DAVIS 2016 data set for training process ?
I would really appreciate it if you can reply to me.

weight of the deconvolutional layer

The weight_filler in the deconvolutional layer is missed and the lr_mult is set to be 0. I wonder whether it means that the weight of the deconvolutional layer is initialized in a default way and the weight is not updated after the initialization. If this is the case, why not update the weight? Any help is appreciated, and looking forward to your early reply.

Parent network training, loss = nan ???

I use the Titan XP to train the parent model. I tried base_lr = 1e-8(default) / 1e-9, but the loss always fluctuates ups and downs, and it became 'nan' at some iteration. Could you tell me how to tackle the problem? I would really appreciate it if you could give a reply.

I0321 21:52:42.315754 11354 solver.cpp:331] Iteration 0, Testing net (#0)
I0321 21:52:44.937382 11354 solver.cpp:219] Iteration 0 (0 iter/s, 2.62483s/20 iters), loss = 116012
I0321 21:52:44.937474 11354 solver.cpp:238] Train net output #0: dsn2_loss = 45369.6 (* 1 = 45369.6 loss)
I0321 21:52:44.937501 11354 solver.cpp:238] Train net output #1: dsn3_loss = 45369.6 (* 1 = 45369.6 loss)
I0321 21:52:44.937510 11354 solver.cpp:238] Train net output #2: dsn4_loss = 45369.6 (* 1 = 45369.6 loss)
I0321 21:52:44.937517 11354 solver.cpp:238] Train net output #3: dsn5_loss = 45369.6 (* 1 = 45369.6 loss)
I0321 21:52:44.937525 11354 solver.cpp:238] Train net output #4: fuse_loss = 45369.6 (* 1 = 45369.6 loss)
I0321 21:52:44.937536 11354 sgd_solver.cpp:105] Iteration 0, lr = 1e-09
I0321 21:53:38.136545 11354 solver.cpp:219] Iteration 20 (0.375968 iter/s, 53.196s/20 iters), loss = 132717
I0321 21:53:38.136634 11354 solver.cpp:238] Train net output #0: dsn2_loss = 30310.2 (* 1 = 30310.2 loss)
I0321 21:53:38.136662 11354 solver.cpp:238] Train net output #1: dsn3_loss = 29715.9 (* 1 = 29715.9 loss)
I0321 21:53:38.136678 11354 solver.cpp:238] Train net output #2: dsn4_loss = 30441.8 (* 1 = 30441.8 loss)
I0321 21:53:38.136693 11354 solver.cpp:238] Train net output #3: dsn5_loss = 30483.2 (* 1 = 30483.2 loss)
I0321 21:53:38.136708 11354 solver.cpp:238] Train net output #4: fuse_loss = 30387.5 (* 1 = 30387.5 loss)
I0321 21:53:38.136724 11354 sgd_solver.cpp:105] Iteration 20, lr = 1e-09
I0321 21:54:30.748098 11354 solver.cpp:219] Iteration 40 (0.380165 iter/s, 52.6087s/20 iters), loss = 107836
I0321 21:54:30.748172 11354 solver.cpp:238] Train net output #0: dsn2_loss = 19628.8 (* 1 = 19628.8 loss)
I0321 21:54:30.748181 11354 solver.cpp:238] Train net output #1: dsn3_loss = 19075.7 (* 1 = 19075.7 loss)
I0321 21:54:30.748188 11354 solver.cpp:238] Train net output #2: dsn4_loss = 19764.4 (* 1 = 19764.4 loss)
I0321 21:54:30.748196 11354 solver.cpp:238] Train net output #3: dsn5_loss = 19837 (* 1 = 19837 loss)
I0321 21:54:30.748219 11354 solver.cpp:238] Train net output #4: fuse_loss = 19558.3 (* 1 = 19558.3 loss)
I0321 21:54:30.748229 11354 sgd_solver.cpp:105] Iteration 40, lr = 1e-09
I0321 21:55:20.153046 11354 solver.cpp:219] Iteration 60 (0.404839 iter/s, 49.4024s/20 iters), loss = 99894
I0321 21:55:20.153127 11354 solver.cpp:238] Train net output #0: dsn2_loss = 10907.5 (* 1 = 10907.5 loss)
I0321 21:55:20.153141 11354 solver.cpp:238] Train net output #1: dsn3_loss = 9222.18 (* 1 = 9222.18 loss)
I0321 21:55:20.153151 11354 solver.cpp:238] Train net output #2: dsn4_loss = 11842.5 (* 1 = 11842.5 loss)
I0321 21:55:20.153169 11354 solver.cpp:238] Train net output #3: dsn5_loss = 11927.8 (* 1 = 11927.8 loss)
I0321 21:55:20.153182 11354 solver.cpp:238] Train net output #4: fuse_loss = 10795.1 (* 1 = 10795.1 loss)
I0321 21:55:20.153194 11354 sgd_solver.cpp:105] Iteration 60, lr = 1e-09
I0321 21:56:07.384881 11354 solver.cpp:219] Iteration 80 (0.423465 iter/s, 47.2294s/20 iters), loss = 44943.5
I0321 21:56:07.384968 11354 solver.cpp:238] Train net output #0: dsn2_loss = 17211.5 (* 1 = 17211.5 loss)
I0321 21:56:07.384996 11354 solver.cpp:238] Train net output #1: dsn3_loss = 11019.4 (* 1 = 11019.4 loss)
I0321 21:56:07.385004 11354 solver.cpp:238] Train net output #2: dsn4_loss = 18592.5 (* 1 = 18592.5 loss)
I0321 21:56:07.385011 11354 solver.cpp:238] Train net output #3: dsn5_loss = 18892.4 (* 1 = 18892.4 loss)
I0321 21:56:07.385018 11354 solver.cpp:238] Train net output #4: fuse_loss = 14495.2 (* 1 = 14495.2 loss)
I0321 21:56:07.385027 11354 sgd_solver.cpp:105] Iteration 80, lr = 1e-09
I0321 21:56:54.658689 11354 solver.cpp:219] Iteration 100 (0.423089 iter/s, 47.2714s/20 iters), loss = 67899.7
I0321 21:56:54.658788 11354 solver.cpp:238] Train net output #0: dsn2_loss = 14982.7 (* 1 = 14982.7 loss)
I0321 21:56:54.658818 11354 solver.cpp:238] Train net output #1: dsn3_loss = 10885.6 (* 1 = 10885.6 loss)
I0321 21:56:54.658833 11354 solver.cpp:238] Train net output #2: dsn4_loss = 15006.9 (* 1 = 15006.9 loss)
I0321 21:56:54.658844 11354 solver.cpp:238] Train net output #3: dsn5_loss = 15270.3 (* 1 = 15270.3 loss)
I0321 21:56:54.658861 11354 solver.cpp:238] Train net output #4: fuse_loss = 12647.6 (* 1 = 12647.6 loss)
I0321 21:56:54.658876 11354 sgd_solver.cpp:105] Iteration 100, lr = 1e-09
I0321 21:57:43.779109 11354 solver.cpp:219] Iteration 120 (0.407183 iter/s, 49.118s/20 iters), loss = 63917.8
I0321 21:57:43.779191 11354 solver.cpp:238] Train net output #0: dsn2_loss = 9679.94 (* 1 = 9679.94 loss)
I0321 21:57:43.779199 11354 solver.cpp:238] Train net output #1: dsn3_loss = 4138.92 (* 1 = 4138.92 loss)
I0321 21:57:43.779206 11354 solver.cpp:238] Train net output #2: dsn4_loss = 9165.52 (* 1 = 9165.52 loss)
I0321 21:57:43.779215 11354 solver.cpp:238] Train net output #3: dsn5_loss = 11711.8 (* 1 = 11711.8 loss)
I0321 21:57:43.779238 11354 solver.cpp:238] Train net output #4: fuse_loss = 5715.27 (* 1 = 5715.27 loss)
I0321 21:57:43.779247 11354 sgd_solver.cpp:105] Iteration 120, lr = 1e-09
I0321 21:58:35.024227 11354 solver.cpp:219] Iteration 140 (0.3903 iter/s, 51.2426s/20 iters), loss = 87141.5
I0321 21:58:35.024309 11354 solver.cpp:238] Train net output #0: dsn2_loss = 4843.74 (* 1 = 4843.74 loss)
I0321 21:58:35.024336 11354 solver.cpp:238] Train net output #1: dsn3_loss = 2672.27 (* 1 = 2672.27 loss)
I0321 21:58:35.024345 11354 solver.cpp:238] Train net output #2: dsn4_loss = 4293.57 (* 1 = 4293.57 loss)
I0321 21:58:35.024353 11354 solver.cpp:238] Train net output #3: dsn5_loss = 5528.3 (* 1 = 5528.3 loss)
I0321 21:58:35.024363 11354 solver.cpp:238] Train net output #4: fuse_loss = 3142 (* 1 = 3142 loss)
I0321 21:58:35.024382 11354 sgd_solver.cpp:105] Iteration 140, lr = 1e-09
I0321 21:59:14.739044 11354 solver.cpp:219] Iteration 160 (0.503615 iter/s, 39.7129s/20 iters), loss = nan
I0321 21:59:14.739117 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 21:59:14.739128 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 21:59:14.739141 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 21:59:14.739157 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 21:59:14.739171 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 21:59:14.739183 11354 sgd_solver.cpp:105] Iteration 160, lr = 1e-09
I0321 21:59:55.145210 11354 solver.cpp:219] Iteration 180 (0.494997 iter/s, 40.4043s/20 iters), loss = nan
I0321 21:59:55.145300 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 21:59:55.145318 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 21:59:55.145329 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 21:59:55.145340 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 21:59:55.145361 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 21:59:55.145376 11354 sgd_solver.cpp:105] Iteration 180, lr = 1e-09
I0321 22:00:36.563983 11354 solver.cpp:219] Iteration 200 (0.482895 iter/s, 41.4168s/20 iters), loss = nan
I0321 22:00:36.564069 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:00:36.564098 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:00:36.564111 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:00:36.564124 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:00:36.564138 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:00:36.564154 11354 sgd_solver.cpp:105] Iteration 200, lr = 1e-09
I0321 22:01:18.799461 11354 solver.cpp:219] Iteration 220 (0.473557 iter/s, 42.2335s/20 iters), loss = nan
I0321 22:01:18.799545 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:01:18.799561 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:01:18.799572 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:01:18.799583 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:01:18.799597 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:01:18.799616 11354 sgd_solver.cpp:105] Iteration 220, lr = 1e-09
I0321 22:01:59.521841 11354 solver.cpp:219] Iteration 240 (0.491153 iter/s, 40.7205s/20 iters), loss = nan
I0321 22:01:59.521919 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:01:59.521931 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:01:59.521968 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:01:59.522012 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:01:59.522033 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:01:59.522073 11354 sgd_solver.cpp:105] Iteration 240, lr = 1e-09
I0321 22:02:42.450592 11354 solver.cpp:219] Iteration 260 (0.465909 iter/s, 42.9268s/20 iters), loss = nan
I0321 22:02:42.450680 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:02:42.450693 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:02:42.450713 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:02:42.450726 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:02:42.450744 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:02:42.450754 11354 sgd_solver.cpp:105] Iteration 260, lr = 1e-09
I0321 22:03:24.126384 11354 solver.cpp:219] Iteration 280 (0.479916 iter/s, 41.6739s/20 iters), loss = nan
I0321 22:03:24.126468 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:03:24.126497 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:03:24.126510 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:03:24.126528 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:03:24.126543 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:03:24.126565 11354 sgd_solver.cpp:105] Iteration 280, lr = 1e-09
I0321 22:04:02.832152 11354 solver.cpp:219] Iteration 300 (0.516742 iter/s, 38.7041s/20 iters), loss = nan
I0321 22:04:02.832226 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:04:02.832234 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:04:02.832247 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:04:02.832253 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:04:02.832259 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:04:02.832267 11354 sgd_solver.cpp:105] Iteration 300, lr = 1e-09
I0321 22:04:43.423032 11354 solver.cpp:219] Iteration 320 (0.492743 iter/s, 40.5891s/20 iters), loss = nan
I0321 22:04:43.423207 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:04:43.423264 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:04:43.423312 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:04:43.423359 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:04:43.423405 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:04:43.423454 11354 sgd_solver.cpp:105] Iteration 320, lr = 1e-09
I0321 22:05:22.978026 11354 solver.cpp:219] Iteration 340 (0.505648 iter/s, 39.5532s/20 iters), loss = nan
I0321 22:05:22.978121 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:05:22.978135 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:05:22.978149 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:05:22.978162 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:05:22.978175 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)
I0321 22:05:22.978186 11354 sgd_solver.cpp:105] Iteration 340, lr = 1e-09
I0321 22:06:02.740044 11354 solver.cpp:219] Iteration 360 (0.503014 iter/s, 39.7603s/20 iters), loss = nan
I0321 22:06:02.740128 11354 solver.cpp:238] Train net output #0: dsn2_loss = nan (* 1 = nan loss)
I0321 22:06:02.740154 11354 solver.cpp:238] Train net output #1: dsn3_loss = nan (* 1 = nan loss)
I0321 22:06:02.740160 11354 solver.cpp:238] Train net output #2: dsn4_loss = nan (* 1 = nan loss)
I0321 22:06:02.740169 11354 solver.cpp:238] Train net output #3: dsn5_loss = nan (* 1 = nan loss)
I0321 22:06:02.740175 11354 solver.cpp:238] Train net output #4: fuse_loss = nan (* 1 = nan loss)

question about Fast Bilateral Solver (FBS).

Hello,
Fast Bilateral Solver method in your model was used to refine edge, I wonder how you employ it in your project. Thank you very much!

parent network training

Hi,
when I train parent network, I got three models: osvos_parent_step1_iter_5000.caffemodel / osvos_parent_step1_iter_10000.caffemodel / osvos_parent_step1_iter_15000.caffemodel; I am confused about the difference between these three models I get and the model OSVOS_parent.caffemodel can be downloaded? Thank you!