min2209/dwt

Deep Watershed Transform for Instance Segmentation

Deep Watershed Transform

Performs instance level segmentation detailed in the following paper:

Min Bai and Raquel Urtasun, Deep Watershed Transformation for Instance Segmentation, in CVPR 2017. Accessible at https://arxiv.org/abs/1611.08303.

This page is still under construction.

Dependencies

Developed and tested on Ubuntu 14.04 and 16.04.

1. TensorFlow www.tensorflow.org
2. Numpy, Scipy, and Skimage (sudo apt-get install python-numpy python-scipy python-skimage)

Inputs

1. Cityscapes images (www.cityscapes-dataset.com).
2. Semantic Segmentation for input images. In our case, we used the output from PSPNet (by H. Zhao et al. https://github.com/hszhao/PSPNet). These are uint8 images with pixel-wise semantic labels encoded with 'trainIDs' defined by Cityscapes. For more information, visit https://github.com/mcordts/cityscapesScripts/blob/master/cityscapesscripts/helpers/labels.py

Outputs

The model produces pixel-wise instance labels as a uint16 image with the same formatting as the Cityscapes instance segmentation challenge ground truth. In particular, each pixel is labeled as 'id' * 1000 + instance_id, where 'id' is as defined by Cityscapes (for more information, consult labels.py in the above link), and instance_id is an integer indexing the object instance.

Testing the Model

1. Clone repository into dwt/.
2. Download the model from www.cs.toronto.edu/~mbai/dwt_cityscapes_pspnet.mat and place into the "dwt/model" directory.
3. run "cd E2E"
4. run "python main.py"
5. The results will be available in "dwt/example/output".

Training the Model

1. Will be available soon.

dwt's Issues

Question about generating WTN's ground truth

It would be great if you could share the code (or talk about) how to generate the ground truth for WTN. More specifically, what does bin 0 correspond to? In the paper it says

In this case, bin 0 corresponds to background or regions within 2 pixels of an instance boundary...

Does this just mean all the pixels outside of the instance are in bin 0? or bin 0 has all the pixels outside the polygon and 2 pixels inside it?

Thanks!

Can you share the training code please ?

This looks like a super cool idea, any update on the full training code?

Where I can get the GT datasets ?

@min2209

InvalidArgumentError: Incompatible shapes: [1,512,1024,3] vs. [1,512,1024,4,1]

Traceback (most recent call last):
File "main.py", line 69, in
outputSavePath="../example/output")
File "/usr/qiaodong/dwt/E2E/forward.py", line 35, in forward_model
keepProb: 1.0})
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
run_metadata_ptr)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
feed_dict_string, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
target_list, options, run_metadata)
File "/usr/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [1,512,1024,3] vs. [1,512,1024,4,1]
[[Node: model_builder/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0, model_builder/ExpandDims)]]

Caused by op u'model_builder/mul', defined at:
File "main.py", line 69, in
outputSavePath="../example/output")
File "/usr/qiaodong/dwt/E2E/forward.py", line 19, in forward_model
model.build(tfBatchImages, tfBatchSS, tfBatchSSMask, keepProb=keepProb)
File "/usr/qiaodong/dwt/E2E/e2e_model.py", line 21, in build
inputData = inputData * ss
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 838, in binary_op_wrapper
return func(x, y, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 1061, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1377, in _mul
result = _op_def_lib.apply_op("Mul", x=x, y=y, name=name)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Incompatible shapes: [1,512,1024,3] vs. [1,512,1024,4,1]
[[Node: model_builder/mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_Placeholder_0_0, model_builder/ExpandDims)]]

issue when generating ground truth for direction map and WTN

I tried to run the batch_generate_cityscapes.m to generate the ground truth mat.
I encountered the following error in generate_GT_cityscapes_unified.m
Error using repmat
Too many input arguments.

Error in generate_GT_cityscapes_unified (line 13)
dir_map = repmat(depth_map, 1, 1, 2);
ps. I used Matlab 2013a
Can someone please tell me what repmat(depth_map, 1, 1, 2); is intended to do?
Also, could someone tell me what 'depth_map', 'dir_map', 'weight_map', 'edge_map' represent? thanks. I suppose dir_map is direction of distance transform.

prerequisite setup before running main.py

Hi, @min2209 ,

According to line 64 -- to 66 in main.py, could you provide an example for sample_list.txt? Furthermore, as PSPNet is needed, is that just git clone https://github.com/hszhao/PSPNet under folder example/ or any further steps needed to perform?

Thanks!

Can You Share the Example Floder

Dear author:
I try to compile the project you share, however there is no "Example" floder. Could you share this floder or reply me the content in the floder?
Thank you!

Compatibility suggestion

tensorflow version : 0.12.0
scikit-image version : 0.13.1

Example to train on own dataset

Hi !
Could you share an example for us to train on our own datasets
Thanks
Mehul

TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

When running main.py as you described in the readme, I get the following error:

Traceback (most recent call last):
File "main.py", line 69, in
outputSavePath="../example/output")
File "/home/mehdi/Desktop/deeplearning/dwt/E2E/forward.py", line 19, in forward_model
model.build(tfBatchImages, tfBatchSS, tfBatchSSMask, keepProb=keepProb)
File "/home/mehdi/Desktop/deeplearning/dwt/E2E/e2e_model.py", line 23, in build
inputData = tf.concat(3, [inputData, tf.expand_dims(ssMask,-1)])
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1111, in concat
dtype=dtypes.int32).get_shape().assert_is_compatible_with(
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 998, in convert_to_tensor
as_ref=False)
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1094, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 217, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 196, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 436, in make_tensor_proto
_AssertCompatible(values, dtype)
File "/home/mehdi/environments/tensorflow-py2.7/local/lib/python2.7/site-packages/tensorflow/python/framework/tensor_util.py", line 347, in _AssertCompatible
(dtype.name, repr(mismatch), type(mismatch).name))
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

Am I missing something?

Can you share the training script?

ValueError: Shape must be rank 4 but is rank 0 for 'model_builder/direction/conv1_1/Conv2D' (op: 'Conv2D') with input shapes: [], [3,3,4,64].

There is something wrong when I run the main.py. It seems like when building the first conv layer the builder could not get the shape of input data (the placeholders' shape is just [ ] ). What should I do to run the code? Thanks.

Hi, Please witch version of tensorflow,numpy,scipy and skimage those you used ?