xing0047/tps

[ECCV 2022] Domain Adaptive Video Segmentation via Temporal Pseudo Supervision

Ph.D. Student on Artificial Intelligence @ Nanyang Technological University

Interested in

• Vision-Language Pre-training
• Unsupervised Domain Adaptation

Pinned

• TPS
• rewrite

tps's Issues

RuntimeError: cuda runtime error (77) : an illegal memory access was encountered

I really enjoy reading your work！！
At the same time, I encountered a problem in the operation of
CUDA_ VISIBLE_ DEVICES=0 python train.py --cfg configs/tps_ viper2city.yml

Have you ever encountered such a problem?
Could you please kindly tell me how to solve this problem? Thank you!
@xing0047 @Dayan-Guan

Error :
Traceback (most recent call last):
File "./tps/scripts/train.py", line 160, in
main()
File "./tps/scripts/train.py", line 157, in main
train_domain_adaptation(model, source_loader, target_loader, cfg)
File "/home/customer/Desktop/ZZ/FMFSemi/TransferLearning/examples/domain_adaptation/video_seg/TPS/tps/domain_adaptation/train_video_UDA.py", line 30, in train_domain_adaptation
train_TPS(model, source_loader, target_loader, cfg)
File "/home/customer/Desktop/ZZ/FMFSemi/TransferLearning/examples/domain_adaptation/video_seg/TPS/tps/domain_adaptation/train_video_UDA.py", line 179, in train_TPS
src_pred_aux, src_pred, src_pred_cf_aux, src_pred_cf, src_pred_kf_aux, src_pred_kf = model(src_img_cf.cuda(device), src_img_kf.cuda(device), src_flow, device)
File "/opt/software/anaconda3/envs/TPS/lib/python3.6/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, **kwargs)
File "/home/customer/Desktop/ZZ/FMFSemi/TransferLearning/examples/domain_adaptation/video_seg/TPS/tps/model/accel_deeplabv2.py", line 153, in forward
pred_aux = self.sf_layer(torch.cat((cf_aux, self.warp_bilinear(kf_aux, flow_cf)), dim=1))
RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/THCGeneral.cpp:371

Compatible with Higher CUDA Versions?

Hi @xing0047 , thanks for open-sourcing your work!

Is the current TPS codebase compatible with higher CUDA versions (for newer PyTorch versions), for example, 11.3?

The current configuration requires torch==1.2.0 torchvision==0.4.0, which is only viable for lower CUDA versions like CUDA 10.0 and 9.2.

I have tried to compile the resample2d package under higher torch versions but it is not successful. It seems highly dependent on torch==1.2.0 torchvision==0.4.0.

Looking forward to your help.

In train_video_UDA.py, line 251, trg_ prob_ warp = warp_ bilinear(trg_prob, trg_flow_warp), if the image flips, but the optical flow does not flip

Hello!
I really enjoy reading your work！！
At the same time, I encountered a problem in the operation of train_video_UDA.py

In line 251 trg_ prob_ warp = warp_ bilinear(trg_prob, trg_flow_warp)，
Variable trg_prob is the prediction of trg_img_b_wk,
and trg_img_b_wk is obtained by trg_img_b based on a certain probability of flip,
but trg_flow_warp does not seem to be flipped,
We consider such a situation,
If trg_img_b_wk is fliped,
trg_flow_warp is not flipped,
Then trg_prob_warp and trg_img_d_st do not seem consistent? Because the image flips, but the optical flow does not flip. Although the trg_pl in line 256~258 is fliped.

Chinese discription of my question:
在第251行， trg_ prob_ warp = warp_ bilinear(trg_prob, trg_flow_warp)，
变量trg_prob是trg_img_b_wk的语义分割预测，
而trg_img_b_wk是由trg_img_b根据一定概率flip得到的，
但 trg_flow_warp似乎没有进行翻转，
我们考虑这样一种情况，
如果trg_img_b_wk经过了flip处理，
那么trg_prob_warp和trg_img_d_st的语义貌似不是一致的？因为图像flip了但光流图没有flip。
尽管在第256行对trg_pl进行了flip操作

A question about the training time and GPU

Hi,
As the highlight claim：“TPS is 3x faster than previous DA-VSN in training while achieves SOTA in domain adaptive video segmentation task.”

I also want to ask a question: what type of GPU is used for TPS and DA-VSN training? How many GPUs are used? How long did they train each other?
Thank you!

Cannot find 'resample2d_cuda.cc' while installing

In step 4 of installation:

resample2d dependency:
python ./TPS/tps/utils/resample2d_package/setup.py build
python ./TPS/tps/utils/resample2d_package/setup.py install

It reports fault that

gcc: error: resample2d_cuda.cc: No such file or directory

This can be fixed by enter this directory TPS/tps/utils/resample2d_package, and run

python setup.py build
python setup.py install

A problem when running the train.py

Hi,

Thanks for your great work!

I met a problem when I run the train.py ,

CUDA_VISIBLE_DEVICES=0 python train.py --cfg configs/tps_syn2city.yml
Traceback (most recent call last):
File "train.py", line 11, in
from tps.model.accel_deeplabv2 import get_accel_deeplab_v2
ModuleNotFoundError: No module named 'tps'

Could you pls give any suggestions?

And The link for downloading optical flow is not accessible, could you pls update them?

Thank you

import resample2d_cuda error occurred

ImportError: /xxxx/TPS/tps/utils/resample2d_package/resample2d_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

how can I fixed it?
The environment：
torch==1.2.0
torchvision==0.4.0
python 3.6

Which pictures of which dataset are used to make the video of [Video Demo]?

Hi,
Good job!!
I want to ask which pictures of which dataset are used to make the video of [Video Demo].
It seems that it is not based on val data of cityscapes dataset，thanks!!
best wishes,
zhe

Question about the preprocess of the Synthia dataset

TPS/tps/dataset/SynthiaSeq.py

Line 22 in 12e09b4

image = image[:-120, :, :]

Hi! I wonder why the Synthia dataset is cropped from (760, 1280) to (640, 1280). I feel a little confused.

Thanks

About the estimated optical flow

Hello @xing0047 , thank you for open-sourcing your work!

I initially tried to use optical flow estimation to get the optical flow files necessary for training, but it seemed complicated.
I wish to experiment with pre-trained optical flow first, but unfortunately, I can't access your link:
https://drive.google.com/file/d/18q6KH-beoBp5jSr1Pl1lMiEcb2te2vxq/view?usp=sharing
https://drive.google.com/file/d/1aOeyBLECPSW_ujMBE9RXKjVhTbhw4L2O/view?usp=sharing
https://drive.google.com/file/d/193uZifde7WiuImwAgshkPTt1Z6zgE3z8/view?usp=sharing
https://drive.google.com/file/d/1USizndlUewVb8Eqh4SV6uNuLCEfV9vzU/view?usp=sharing
These links are all inaccessible. Could you provide a new access link?

Question about the dataset SEQS-04-DAWN

A very nice job! But I also encountered the same problem, and the SEQS-04-DAWN folder I downloaded has two subfolders, RGB and GT, and the RGB folder has two angles, Stereo_Right and Stereo_Left,

And the folders of both angles contain four folders: Omni_B, Omni_F, Omni_L, and Omni_R. What kind of files are used in the document?

In addition, each file only contains 850 pictures, which does not seem to correspond to the 8000 frames mentioned in the article. I would like to ask for your advice, thank you very much! !