yzou2/crst

Code for <Confidence Regularized Self-Training> in ICCV19 (Oral)

crst's Issues

About dataset/list/gta5/train.lst & val.lst

Hi.
I am wondering why 19,984 images are in the gta5 train and val list, rather than 24,966 images.
Why 4,982 images are removed?
Thanks.

ResNet-38 for CRST

Hi, Thank you for your great work and contribution.
I am in the middle of reproducing your work with proposed backbones.
Is there a PyTorch based initial ResNet-38 model that is pre-trained on GTA and SYNTHIA to get the paper results?
Thank you again, and I hope I get a positive response.

Cityscape and GTA Dataset

Hi @yzou2 ,

Really nice work!
Just wanted to make sure I am working with the right dataset, we need :
1)gtFine_trainvaltest and leftImg8bit_trainvaltest from cityscapes and
2)all the 10 parts from the GTA dataset (24966 images)?

Implementation details for Synthia

Hi,

I'm trying to get the Synthia transfer to work, and I cant see to do so. I followed the hyperparameter settings in #14 and using the source model in #24 but I cant seem to have any luck in getting close to the reported numbers. I'm currently getting about 4-5% lesser than the reported ones.

I'm using the following resizing too following #18, i use

label = cv2.resize(label, (2048, 1024), interpolation=cv2.INTER_NEAREST)                                                                                                                       
image = cv2.resize(image, (2048, 1024))   

I cant seem to exceed 38% mIoU (reported 42+). Do you know what might be missing?

About train_ClsConfSet.lst

In your code, train_ClsConfSet.lst only contains 505 images. Is there any reason you didn't used 2975 images for pseudo label generation?

about the Eq.1 in the paper

why we use p(k|x_t, w) / λ_k in the target domain instead of only p(k|x_t, w). In my opinion, λ_k is used for select confidence threshold as in Eq.4

MRL2 and MRENT

Hi @yzou2 ,

Will you be releasing MRL2 and MRENT codes for baseline?
I would like to reproduce the experiments from your paper.

Thanks!

ValueError: Unknown format code 'f' for object of type 'str'

Hello！I successfully ran the first round, but encountered such a problem in the next round, what is the reason? Why can the first round succeed, but the following will not work? What is the reason for multiple rounds?Thank you so much！

Reproducing the VisDa-17 results

Hi,

I am interested in reproducing the results for VisDa-17 benchmark. I couldn't find any instructions on how to do this in the repository.

Reproducing the numbers given

Hi,
I'm trying to run your code to reproduce the results. However, I'm running into some issues. Using the same packages (pytorch, python versions), I'm getting the following the results.

Method	Weight	Result
lr_ent	0.25	44.59%
cbst		2.54%
mr_weight_kld	0.1	2.86%

I've used the defaults prescribed in the codebase itself.

Can you tell me what might be doing wrong?

Cityscapes and GTA dataset

Hi @yzou2 ,

Really nice work!
Just wanted to make sure that I am working with the right dataset. We need=
1)leftImg8bit_trainvaltest and gtFine_trainvaltest from cityscapes and
2)all 10 parts from GTA dataset (24966 images)?

Minor problem

CRST/crst_seg.py

Line 663 in 0374db8

weighted_prob = pred_prob/cls_thresh

Division by thresholds first does have a very small chance to change the following argmax results.
e.g. A pixel with 2 classes has softmax result as [0.89, 0.11], but the thresholds so happens to be [0.9, 0.09], then it will satisfy weighted_prob > 1, but it is not even predicted as class 2.
Maybe this is indeed the original intent of the paper, but I'll just point it out here.
I call this minor problem because the example above is very unlikely to happen in actual training.

Question about train_ClsConfSet.lst

Hello, very impressive work!
I have a question about the train_ClsConfSet.lst in CRST/dataset/list/cityscapes/train_ClsConfSet.lst.
In this training list, there are only 505 lines while the whole cityscapes training list should be 2975 lines. Would you mind explaining how do you get this list? Were the results in your paper produced by this subset? Have you ever try what would happen if you use the whole set?
Thank you in advance~

How to train our own dataset?

I want use synthetic dataset as source dataset ,and I change the path and train list.
File "/home/qiu/下载/CRST-master/deeplab/datasets.py", line 178, in getitem
img_h, img_w = label.shape
ValueError: too many values to unpack
SO is the dataset must be the same size as gta5 or some params need change?

VGG code for cbst

Thank you for the great work. Can you provide the code for VGG16? Because I want reproduce the results in your paper. Thank you.

Loss explodes

When training with generated pseudo label, loss explodes.
I run with hyperparameters you provided for SYNTHIA (#14)
and set my environment to required version.
Can you give any advice to solve this problem?

Questions about selecting 0 or $\hat{y}_t$ ?

Hi, thanks for your shared code.
But, I cannot find the code to select between $\hat{y}_t$ or 0 by checking which leads to a lower cost. Could you show the corresponding position ?
Thanks.

sorry to trouble

Anyone reproduced the results with pytorch0.4.0, python 3.6.9, OpenCV 4.4 ?

Thanks for checking this issue.
I attempts to install python2.7 + pytorch0.4.0 on my machines, BUT FAILED EVERYTIME ...
Then I wonder:

1. what the difference in python3.6.9 + opencv 4.4 from python2.7 + opencv 3.2?
2. since python2.7 retires in 2020, then it is necessary for CRST can reproduce the results in python 3.6.9 ... How can we make it?

Hyper-parameter for SYNTHIA dataset

Hi, Sorry for the successive question and Thank you again for your work.

Is it possible for you to let me know the hyper-parameters for the SYNTHIA dataset for paper results, specifically "init_src_port" and "Input_size" which are commented "for GTA" besides?

Thank you and sorry for interrupting again.

Question about GTA5 dataset training list

Hi, thanks for your impressive work.

I have a question regarding the list of GTA5 training set. As far as I know, GTA5 dataset has about 25k images, while in the training list you provide at here, there are only about 20k lines, which means that about 5k images are filtered out for training. Would you like to spend some time explaining how do you get this list and why several images are missing? Thanks in advance!

no mix domain

I wanted to do the cross entropy loss of the source domain and the target domain respectively, and then sum them up，backword total loss. But nan always happens. What's the matter？Due to memory constraints，i set batch_size = 1.

Implementation detail request for classification task

Hi,

For implementation detail for Office-31 and Visda17, do you also follow those two steps?

1. Pretraining on the source dataset
2. Self-training on the target dataset

If so, could you please provide how many epochs you train for source dataset in step 1) for two datasets and how many epochs you train for step 2) ?

Best,
Chang

strange result??

When I run your crst.seg with src-model using mrkld.sh, after training with pseudo label, iou drop drastically.

Used Models

Hi,
After reading the paper I was expecting a ResNet-38 based implementation as it yielded the better results than DeepLabV2 but if I am not mistaken there is only a DeepLabV2 based implementation. Am I missing something?

If I am not mistaken, are the logged results found on this repository were generated with DeepLabV2 training?

Thanks for this contribution.

which mode for self training?

CRST/crst_seg.py

Line 380 in f7e4df0

if args.is_training:

Thanks for your sharing! I would like to ask if you use evaluation mode here intentionally? If yes, I would like to ask the reason for this. To my understand here the condition is false and the evaluation mode is used, since I haven't seen the setting about "--is-training" in shell files.

Problem when reading SYNTHIA labels

CRST/deeplab/datasets.py

Line 584 in f7e4df0

label = np.array(Image.open(datafiles["label"]))

If you use this method to read SYNTHIA ground truth label, the output will be wrong.
And this should work
label = np.asarray(imageio.imread(datafiles["label"], format='PNG-FI'))[:,:,0]

problem about new dataset label convert

Hello, I trained a model based on synthia dataset and the label noted in labels_synthia.py . But when I evaluate the model with cityscapes and the label from labels.py , I found the result is absolutely wrong. Do you know what step is wrong?

By the way , I change the datasets.py as follow to read the label in synthia because cv2 couldn't read the label in one channel.

SYNTHIA_label_map = {3: 0, 4: 1, 2: 2, 21: 3, 5: 4, 7: 5, 15: 6, 9: 7, 6: 8, 1: 9, 10: 10, 17: 11, 8: 12, 19: 13, 12: 14, 11: 15}
#image_size = (640, 360)
def get_label_set(input):
reshape_list = list(np.reshape(input,(-1,)))
label_set = set(reshape_list)
return label_set

def read_SYNTHIA_label(label_path, kv_map):
raw_label = cv2.imread(label_path,-1)
raw_label_p = raw_label[:, :, -1]
label = raw_label_p
label_copy = 255 * np.ones(label.shape, dtype=np.float32)
for k, v in kv_map.items():
label_copy[label == k] = v #others are turned to 255
return label_copy
label = np.array(read_SYNTHIA_label(datafiles["label"], SYNTHIA_label_map), dtype=np.uint8)

Could you release the code on office-31 dataset.

performance problems with pytorch1.2.0

I meet a severe drop of qualities of pseudo-labels from round 0 to round 1. do you have any idea about this problem?

Regularizer weight in MRKLD

Hi,

In your paper, the regularizer weight mr_weight_kld is set as 0.1 for MRKLD. But I found when calculating kld distance, you multipled the logsoftmax with another weight reg_weights which is also 0.1. So the overall weight for the regularization term is 0.1*0.1=0.01.

May I know what is this 'reg_weighs'

CRST/crst_seg.py

Line 858 in f7e4df0

kld = torch.sum( -logsoftmax_val/num_class*reg_weights )

CRST/crst_seg.py

Line 861 in f7e4df0

reg_ce = ce/valid_num + (mr_weight_kld*kld)/valid_reg_num

How many iterations of source only training for SYNTHIA dataset ?

I'm trying to reproduce the SYNTHIA -> Cityscape task of CBST paper while GTA5 -> CityScape seemd well, using ResNet38.

But after about 30k iterations of the training in source(SYNTHIA), the mIOU of below stage is just about 3.8%.

CRST/crst_seg.py

Lines 326 to 327 in f7e4df0

	conf_dict, pred_cls_num, save_prob_path, save_pred_path = val(model, device, save_round_eval_path, round_idx, tgt_num,
	label_2_id, valid_labels, args, logger)

Is this mIOU the cause of insufficient source-only training ?

Problem reproducing CRST-MRKLD result

Hi,
I ran the mrkld.sh script with the right data paths. I did not change anything else.
The training process seems to have something wrong, The mIoU is going down drastically in every round all the way down to around 4% success.

Is there anything else needed to ne done on order to reproduce the 47% result?

Could you release the training list for Synthia?

I noticed that you filtered out training images for GTA5 in which sidewalks dominate the bottom region. And the layout gap between Synthia and Cityscapes is more obvious.

Memoryerror

Hello,I am very interested about your project.But when I started cbst with gta2cityscapes,something was wrong.It shows cuda out of memory.And when I set batch_size =1,--mine-chance 0.0,another memoryerror happens as follow:
traceback (most recent call last):
File "crst_seg.py", line 954, in
main()
File "crst_seg.py", line 327, in main
label_2_id, valid_labels, args, logger)
File "crst_seg.py", line 464, in val
output = 0.5 * ( output + softmax2d(interp(output2)).cpu().data[0].numpy()[:,:,::-1] )
MemoryError
MY cpu Intel(R) Core(TM) i3-8100 CPU @ 3.60GHz
gpu GeForce GTX 1080ti
cuda8,pytorch 0.4.0
python 2.7.16

Implementation of Spatial Prior

Hi,
Thank you for your excellent work.

I'm currently trying to implement spatial prior(SP) as presented in your CBST paper (at this moment, only in the pretraining), since it's apparently not implemented in this pytorch version.
After applying the SP, I guess the resulting output scores will be a few orders of magnitude less than those without applying the SP, which results in so small gradient that the training doesn't proceed well.

I've come up with several measures for this:

1. Simply multiplying the resulting output by a random number (like, 10e+4) before calculating cross entropy loss.
2. Normalize the output values of a pixel over the classes (via softmax or simply dividing by the sum)
3. Increase the learning rate.

I looked it up on google, but couldn't find exact answer for my question.
Could you give me an advice on it?
Sorry for the silly question.

Thanks for your work again, and I'm happy to hear from you.

Resizing cityscape

Hi, The work is impressive. I have a queation about resizing cityscape. Should I resize cityscape into 1052x1914 just same as GTA5 do. According to the guidance you only inform us to resize GTA5.

Trained model for round1 and round2 in CBST

Hi, thanks for your code. Could you provide the trained model for round1 and round2 in CBST? I want to use those models to generate pseudo labels. Thanks!!