xtudbxk/dsrg-tensorflow

a tensorflow version for DSRG (Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing)

dsrg-tensorflow's Issues

Transfer Learning of your model

@xtudbxk .The model is for 21 categories.Can u please suggest methods to convert it to a 2 category model?

Loss calculation is too slow

Hi,

First of all, thank you very much for providing the tf implementation.
I am trying to recreate the results in PyTorch, but I have very slow training time due to the consistency loss computation, which uses pydensecrf, and it takes a lot of time for a moderately sized images (~ 356).
Do you have any recommendations on how to speed things up, and can you please provide the time it took to train the network for in your case.

Thank you very much.

can't download localization_cues-cal.pickle

Hi,I encountered difficulty in download this localization_cues-cal.pickle file .I can't open the CAM url in DSRG[https://github.com/speedinghzl/DSRG], because I'm in China and I can't visit Google...
Could you provide me this file maybe in the form of BaiduNetDisk?
Thank you very much!!!

Meet Runtime Warning lead to constrain_loss:0.000000

After training 250 iterations, I meet a problem of RuntimeWarning: divide by zero encountered in log

unary = -np.log(feat)

after that, constrain_loss=0.000000, how could I colve this problem?

test

hi @xtudbxk , thanks for your code,I have a question that how can I test my own image after train?

question about model

Hi, there some questions I'd like to ask you

1. what is the init.model(vgg16_deeplab_aspp.npy) used for, is used to pretrain?
2. Do you have the code for the pytorch version of this paper?Thanks!

About the initial weights

What are the initial weights "vgg16_deeplab_aspp.npy" from? Why does the state_dict contain fc8-SEC_1, fc8_SEC_2, fc8-SEC_3, fc8-SEC_4? Is that correct the initial weights initialise layers from Conv1 to fc7, and fc8_1, fc8_2, fc8_3 and fc8_4 are trained from scratch?

About the miou

Hello, for the init.model vgg16_deeplab_aspp.npy, is it converted from this link http://liangchiehchen.com/projects/DeepLabv2_vgg.html? I noticed that the miou of deeplab v2 can reach above 0.6 in the paper of deeplab v2 in the PASCAL VOC 2012. However, after trained using the DSRG model, the miou decrease to about 0.57. Why did the miou decrease using the DSRG model? Is there something wrong with my understandings? Thank you very much!

Do you have a pytorch version? thanks

train on multiple gpus

Thanks very much for your code.

The script accepts arguments like 0,1,2,3 to specify multiple gpus.
However, only 1 gpu is actually used for computation.

Could you explain how to train with multiple gpus?

Thanks!

Error while training the model

I am trying to reproduce the results but I am getting the below error while training the model.

Traceback (most recent call last):
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
    return fn(*args)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot batch tensors with different shapes in component 1. First element had shape [321,321,1] and element 2 had shape [321,321,3].
         [[{{node IteratorGetNext}} = IteratorGetNext[output_shapes=[[?,321,321,3], [?,321,321,1], [?,21], [?,41,41,21], [?]], output_types=[DT_FLOAT, DT_UINT8, DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "DSRG.py", line 407, in <module>
    dsrg.train(base_lr=lr,weight_decay=5e-4,momentum=0.9,batch_size=batch_size,epoches=epoches)
  File "DSRG.py", line 377, in train
    self.sess.run(self.net["accum_gradient_accum"])
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot batch tensors with different shapes in component 1. First element had shape [321,321,1] and element 2 had shape [321,321,3].
         [[node IteratorGetNext (defined at DSRG.py:347)  = IteratorGetNext[output_shapes=[[?,321,321,3], [?,321,321,1], [?,21], [?,41,41,21], [?]], output_types=[DT_FLOAT, DT_UINT8, DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]

Caused by op 'IteratorGetNext', defined at:
  File "DSRG.py", line 407, in <module>
    dsrg.train(base_lr=lr,weight_decay=5e-4,momentum=0.9,batch_size=batch_size,epoches=epoches)
  File "DSRG.py", line 347, in train
    data_x,data_y,data_tag,data_cues,id_of_image,iterator_train = self.data.next_batch(category="train",batch_size=batch_size,epoches=-1)
  File "/home/ssindhu/winter2020/DSRG-tensorflow/pythonlib/dataset_DSRG.py", line 109, in next_batch
    img,gt,tag,cues,id_ = iterator.get_next()
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 421, in get_next
    name=name)), self._output_types,
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2069, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/ssindhu/deeplab_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Cannot batch tensors with different shapes in component 1. First element had shape [321,321,1] and element 2 had shape [321,321,3].
         [[node IteratorGetNext (defined at DSRG.py:347)  = IteratorGetNext[output_shapes=[[?,321,321,3], [?,321,321,1], [?,21], [?,41,41,21], [?]], output_types=[DT_FLOAT, DT_UINT8, DT_FLOAT, DT_FLOAT, DT_STRING], _device="/job:localhost/replica:0/task:0/device:CPU:0"](IteratorV2)]]

an error

THX for UR work！
but I got an error

Traceback (most recent call last):
File "DSRG.py", line 411, in
dsrg.train(base_lr=lr,weight_decay=5e-4,momentum=0.9,batch_size=batch_size,epoches=epoches)
File "DSRG.py", line 371, in train
self.saver["lr"].save(self.sess,os.path.join(self.config.get("saver_path","saver"),"lr-%f" % base_lr),global_step=i)
AttributeError: 'DSRG' object has no attribute 'saver'

about data

hi
the download rate of data ' localization_cues-cal.pickle' in BaiduNetdisk is too low, so may I request some other way like qq mail to get that. My email: [email protected]. Thx.

loss details

hi @xtudbxk , could you provide some detailed infomation about the loss curve of your result which is 0.564?
when I ran the code, the loss of epoch 0 is 1.0 and then it increased to 12, after that it decreased gradually and finally vibrated around 1.3, and MIOU on validation set is only 52.90(use single scale). Could you provide some suggestions? thanks~

question about localization_cues-cal.pickle

How can I download it ? Does it mean download CAM seed and put it in training/localization_cues folder?

Retrain to boost performance

Hi,

Have you tried the retrain step, which can boost the performance by around 1.5%, as mentioned in the original DSRG paper?

I tried to implement the retrain step based on your code, but I got worse results compared to those without retrain. Sad..............................................................................................................

关于readme中附带的trained model

您好，我想问下您在readme文档中附带的trained model是指已经完全训练到达到DSRG原论文中一样的效果吗？？因为我用您给的trained model和提供的用于评估的程序，跑出来的图分割效果跟DSRG论文中的分割效果不一样，所以想问一下是模型问题还是？？

Question about epoch

@xtudbxk Hi, why epoch add with 300 each loop in the train function, is this a spell mistake?Thanks.

DSRG-tensorflow/DSRG.py

Line 393 in 9e5a832

epoch += 300

How to generate pickle files of other datasets

I want to use my own dataset to generate the corresponding pickle file, but I don't know how to do it. Can you help me to solve it

Evaluation code

Hi, @xtudbxk. Could you provide the evaluation code, thanks~

Question

How could I get localization_cues-cal.pickle and init.model？ @xtudbxk

DSRG_ResNet101

Hi, thank you so much for your efforts to release such a beneficial repo.

DSRG-deeplab-aspp-vgg16 is workable. I tried to replace the vgg16 in your code with ResNet101 following https://github.com/zhengyang-wang/Deeplab-v2--ResNet-101--Tensorflow/blob/master/model.py, but I failed. The total loss seemed to be unchanged and the constrain loss was near 0 from the beginning. I couldn't figure out the problem. I wonder if you have a plan to release a DSRG-deeplab-aspp-resnet101 tf-version? Or maybe could you please kindly give me some tips how to adapt deeplab-aspp-resnet101 to your current framework? I'm looking forward to your reply. Many thanks!!!

training log: (I have edited: unary = -np.log(feat + 1e-5))
Restored model parameters from C:/Tensorflow/DSRG-tensorflow/resnet_v1_101.ckpt
start_time: 1565056349.789119
config -- lr:0.001000 weight_decay:0.000500 momentum:0.900000 batch_size:4.000000 epoches:32.000000
epoch:0.000000, iteration:0.000000, total_loss:27.615696
seed_loss:17.612474,constrain_loss:0.002580
epoch:0.003781, iteration:10.000000, total_loss:27.492531
seed_loss:18.424877,constrain_loss:0.000213
epoch:0.007561, iteration:20.000000, total_loss:27.569458
seed_loss:18.424877,constrain_loss:0.000341
epoch:0.011342, iteration:30.000000, total_loss:27.570873
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.015123, iteration:40.000000, total_loss:27.570740
seed_loss:18.424877,constrain_loss:0.000137
epoch:0.018904, iteration:50.000000, total_loss:22.964874
seed_loss:18.424877,constrain_loss:0.000025
epoch:0.022684, iteration:60.000000, total_loss:27.569254
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.026465, iteration:70.000000, total_loss:25.265747
seed_loss:16.122267,constrain_loss:-0.000000
epoch:0.030246, iteration:80.000000, total_loss:25.264816
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.034026, iteration:90.000000, total_loss:27.566475
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.037807, iteration:100.000000, total_loss:25.635326
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.041588, iteration:110.000000, total_loss:27.564554
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.045369, iteration:120.000000, total_loss:27.563591
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.049149, iteration:130.000000, total_loss:27.562630
seed_loss:13.819657,constrain_loss:-0.000000
epoch:0.052930, iteration:140.000000, total_loss:26.324894
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.056711, iteration:150.000000, total_loss:27.560703
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.060491, iteration:160.000000, total_loss:27.559740
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.064272, iteration:170.000000, total_loss:27.558777
seed_loss:18.424877,constrain_loss:-0.000000
epoch:0.068053, iteration:180.000000, total_loss:25.255207
seed_loss:18.424877,constrain_loss:-0.000000

number of elements does not match.

when I run the project something wrong happened, information as follows:

`tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot add tensor to the batch: number of elements does not match. Shapes are: [tensor]: [321,321,3], [batch]: [321,321,1]
[[node IteratorGetNext (defined at /workbench/DSRG-tensorflow-master/pythonlib/dataset_DSRG.py:109) ]]

Errors may have originated from an input operation.
Input Source operations connected to node IteratorGetNext:
IteratorV2 (defined at /workbench/DSRG-tensorflow-master/pythonlib/dataset_DSRG.py:108)

Original stack trace for 'IteratorGetNext':
File "/workbench/DSRG-tensorflow-master/DSRG.py", line 409, in
dsrg.train(base_lr=lr,weight_decay=5e-4,momentum=0.9,batch_size=batch_size,epoches=epoches)
File "/workbench/DSRG-tensorflow-master/DSRG.py", line 349, in train
data_x,data_y,data_tag,data_cues,id_of_image,iterator_train = self.data.next_batch(category="train",batch_size=batch_size,epoches=-1)
File "/workbench/DSRG-tensorflow-master/pythonlib/dataset_DSRG.py", line 109, in next_batch
img,gt,tag,cues,id_ = iterator.get_next()
File "/anaconda3/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py", line 426, in get_next
output_shapes=self._structure._flat_shapes, name=name)
File "/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gen_dataset_ops.py", line 1947, in iterator_get_next
output_shapes=output_shapes, name=name)
File "/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
`
what should i do？
thank u

about init model

1.想问vgg16_deeplab_aspp.npy是从 http://liangchiehchen.com/projects/DeepLabv2_vgg.html 这个链接里面的init.caffemodel转化得到的吗，这个init.caffemodel是只使用imagenet训过的吗，因为最后一层是21分类，有一些怀疑是训练了分割任务得到的模型权重
2.想问您使用过 https://github.com/speedinghzl/DSRG 里面给的vgg16_20M_mc.caffemodel进行初始化吗
谢谢

Question about provided pretrained weights "vgg16_deeplab_aspp.npy"

Hi, Thank you for preparing the tensorflow DSRG model. In your mark down, it was mentioned that there is the pretrained model which can be downloaded and loaded into the DSRG model. However, I am wondering is this model pretrained by imagenet (for normal classification task) or by Voc2012 (for semantic segmentation). I am not quite sure about the supervision, as the DSRG is the model designed for weakly supervised segmentation, I just want to make sure how much supervision provided by the pretrained weight. . Thanks.

About Loss

I tried to implement in pytorch , but during training , i met a problem that the BalancedSeedLoss will make softmax probability of one in all position , and it does not work in fg area ,and the loss will keep 8~9 . Have you met this problem?
how could I solve this problem? thx.

Error

Thank u for your work so much
It helps me a lot,but i have encountered an error

W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /data/VOCdevkit/VOC2012/JPEGImages/2008_002175.jpg; No such file or directory

W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: /data/VOCdevkit/VOC2012/SegmentationClassAug/2008_002175.png; No such file or directory

I don't know why the error occurs

DSRG training

We are using Segmentation classes of Pascal VOC to generate Augmented images which inturn is used totrain the model which means that we are using fully supervised data isn't it??So how is the model weakly supervised?Please let me know

Missing checkpoint in trained_model_for_dsrg_tf.tar.gz

Hello! Your work is amazing! However, the checkpoint file is not included in the trained_model_for_dsrg_tf.tar.gz. Without it, I can not restore Tensorflow model. Could you please upload it? Thank you very much!