linjieyangsc/video_seg

OSMN: One-Shot Modulation Network for Semi-supervised Video Segmentation

video_seg's Issues

pytorch version

Hello!Is there an implementation version of pytorch?Thank you!

Only 67.4 on davis16 val after training the per-trained model(train_coco) on davis17.

I did the stage 2 training with your official model(train_coco.data-00000-of-00001,train_coco.meta,train_coco.index),but the performance on davis16 declined from 72.2 to 67.4.
The parameters are as follows:
Namespace(aligned_size=None, base_model='osvos', batch_size=4, bbox_sup=False, bbox_valid_ratio=0.2, crf_postprocessing=False, data_aug_scales=[0.5, 0.8, 1], data_path='./database/DAVIS', data_version=2017, display_iters=20, fix_bn=True, gpu_id=0, im_size=[854, 480], label_valid_ratio=0.003, learning_rate=1e-06, max_to_keep=2, mean_value=[104, 117, 123], mod_early_conv=False, model_save_path='/models/MODEL_SAVE_PATH_FT/my_train_full', num_loader=1, only_testing=False, random_crop_ratio=0.0, randomize_guide=True, result_path='/results/DAVIS-2017-my_train', resume_training=True, save_iters=1000, save_score=True, scale_value=1.0, sg_center_perturb_ratio=0.2, sg_std_perturb_ratio=0.4, test_split='val', train_seg=True, training_iters=50000, use_full_res=False, use_image_summary=True, use_original_mask=False, use_spatial_modulator=True, use_visual_modulator=True, vg_color_aug=False, vg_keep_aspect_ratio=False, vg_pad_ratio=0.03, vg_random_crop_ratio=0.1, vg_random_rotate_angle=10, whole_model_path='/models/MODEL_SAVE_PATH/train_coco')
Wait for your reply.Thank you.

What is the expected performance on Youtube-VOS?

Can you provide the expected performance on Youtube-VOS dataset? That would be really helpful for me to diagnose my problem: currently, I got quite low performance (below 45 on dev set). I think I must do sth wrong.

About the performance on Davis2017

@linjieyangsc Thank you very much for sharing. And I have the question about the performance on Davis2017.

After fine-tuning the network on DAVIS2017 (step 2), the performance becomes worse. I am a bit confused. And It seems that you do not report the result in your paper.

Thanks again!

Cannot load provided pretrained model to evaluate DAVIS dataset

I intend to load your pretrained model of DAVIS dataset for evaluation. However, I get errors when trying to run the command followed by your README file instruction.

Caused by op u'save/RestoreV2', defined at:
  File "osmn_train_eval.py", line 198, in <module>
    osmn.test(dataset, args, checkpoint_path, args.result_path, config=config, batch_size=1)
  File "/home/ad/nvtu/video_seg/osmn.py", line 815, in test
    saver = tf.train.Saver([v for v in tf.global_variables() if '-up' not in v.name]) #if '-up' not in v.name and '-cr' not in v.name])
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1284, in __init__
    self.build()
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1296, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1333, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 781, in _build_internal
    restore_sequentially, reshape)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 400, in _AddRestoreOps
    restore_sequentially)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 832, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
    op_def=op_def)
  File "/home/ad/anaconda3/envs/maskrcnn/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1740, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

DataLossError (see above for traceback): Unable to open table file pretrained_model/ft_davis.data-00000-of-00001: Data loss: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]
	 [[Node: save/RestoreV2/_161 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_154_save/RestoreV2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]()]]

My command is:

$ python osmn_train_eval.py --data_path DAVIS --whole_model_path pretrained_model/ft_davis.data-00000-of-00001 --result_path results --only_testing --data_version 2017 --gpu_id 0

What happen and what do I need to use your model? Thank you

Question about spatial modulator

Hello. The paper says that beta_c is two dimensional matrix, but while modulation
beta_c = ~gamma_c m + ~beta_c, where "~gamma_c and ~beta_c are the scale-and-shift parameters for the c-th channel, respectively". Before these words there was nothing said about channels and I thought that "c" meant just "conditional" and that it was a shift for all the channels.
Have some troubles to sort the things out with the code. Couldn't you explain such details of spatial modulation?

low performance with provided model

@linjieyangsc With provided model and evaluation script, I get very low testing performance on davis-16 (stage1: 0.28, stage 2: 0.57).
Could you provide the results (davis-16 and davis-17) used in your paper?

Error while evaluating on DAVIS2017

while running same command i am having some other error

Traceback (most recent call last):
File "osmn_train_eval.py", line 198, in
osmn.test(dataset, args, checkpoint_path, args.result_path, config=config, batch_size=1)
File "/home/ai/video_seg/osmn.py", line 318, in test
guide_images, gb_images, images, save_names = dataset.next_batch(batch_size, 'test')
File "/home/ai/video_seg/dataset_davis.py", line 209, in next_batch
guide_label = Image.open(sample[1])
File "/home/ai/anaconda3/envs/tensorflow2.7/lib/python2.7/site-packages/PIL/Image.py", line 2634, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 20] Not a directory: './DAVIS/Annotations/480p_split/bike-packing/00031.png/00000.png'

I am running command
python osmn_train_eval.py --data_path ./DAVIS --whole_model_path pretrained_model/ft_davis/ft_davis --result_path ./results --only_testing --data_version 2017 --gpu_id 0

Obtained Stage 2 evaluation results on DAVIS2017

Thank you for your work, I would like to ask questions about the score results.
The Stage 2 model obtains mIoU 49 on DAVIS 2017
The model is official model,but I couldn't get the results of the experiment. I did it exactly as indicated on github. If I didn't notice you, please help me correct it.
Thank you!

Implement three measures

Hello! I am a newcomer to learning video segmentation. Do you implement three measures:region similarity,contour accuracy and temporal instability? Thank you !

Hi! I have some questions regarding the guide image used to feed the visual modulator. Where is the guide image formed? The guide image only includes the mask of one instance and a black background or does the guide image consists on the mask of different annotaded objects in the image? Thank you in advance!

where is osmn_train_eval_ytvos.py?

Hello,

Thanks for providing OSMN code.

I want to use Youtube VOS train and eval code, but can not find.

Any comments?