vt-vl-lab / fgvc Goto Github PK
View Code? Open in Web Editor NEW[ECCV 2020] Flow-edge Guided Video Completion
License: Other
[ECCV 2020] Flow-edge Guided Video Completion
License: Other
Can this technology apply to remove the real-time video? on-line?
Greetings! Thanks for sharing your impressive work.
I was trying to find the non-local flow computation and forward-backward flow consistency check in your code. But it seems that the 'NonLocal' option is disabled by default and the 'consistencyThres' is set to inf which means it won't filter out any outliers. The results still seem reasonable, though. Could you please verify if my interpretation is correct and give me some guidance on how to tune these parameters?
Cheers,
Lao
I saw a parameters in argparse called 'alpha', What does the params alpha stand for?
Thank you for this great project! My GPU is 2080ti with CUDA10.2. Therefore I'm wondering if it is gonna support windows os somehow in the future?
Thank you again!
What would the mask input be? An alpha mask where white is the object to be removed?
Hi! I've problems to elaborate 500 images because I go in out of memory of ram. Moreover I tried with 2 images and It went in OOM of VRAM. Maybe resolution must be lower?
I'm running it on Google Colab pro (Max ram settings)
Processing 256 frames at 640x480
Calculating backward flow 146 <---> 147
Finish flow prediction.
Completing backward flow 118 <---> 119tcmalloc: large alloc 1091174400 bytes == 0xf3156000 @ 0x7fe0f8733001 0x7fe0eda164ff 0x7fe0eda66ab8 0x7fe0eda6abb7 0x7fe0edb09003 0x50a4a5 0x50cc96 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281 0x4b0dc0 0x7fe0f832ebf7 0x5b259a
tcmalloc: large alloc 1639604224 bytes == 0x7fdf0c268000 @ 0x7fe0f87311e7 0x7fe0eda1641e 0x7fe0eda66bdb 0x7fe0eda66c78 0x7fe0edab7c39 0x7fe0edb05f70 0x7fe0edb06a6b 0x566fae 0x59fd0e 0x7fe0eda53ea7 0x50a12f 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281
tcmalloc: large alloc 1639604224 bytes == 0x7fde69622000 @ 0x7fe0f87311e7 0x7fe0eda1641e 0x7fe0eda66bdb 0x7fe0eda66c78 0x7fe0edab7c39 0x7fe0edb05f70 0x7fe0edb06a6b 0x566fae 0x59fd0e 0x7fe0eda53ea7 0x50a12f 0x50beb4 0x507be4 0x509900 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x5095c8 0x50a2fd 0x50beb4 0x507be4 0x50ad03 0x634e72 0x634f27 0x6386df 0x639281
bash: line 7: 325 Killed python3 video_completion.py --mode video_extrapolation --path ../demo/papa --outroot ../result/tennis_extrapolation --W_scale 2 --seamless
Thanks for sharing your impressive work!
I was trying to find the non-local flow computation, but how to get these parameters of videoNonLocalFlowF and videoNonLocalFlowB in get_flowNN.py? Could you please give me some guidance for these parameters?
Hi,
Thanks for the great work! I met some problems about the cuda version. For RAFT, the requirement version of cuda is 10.1. However, for environment FGVC, cuda >10 is not supported for pytorch 0.4. It will be a littile incovenient if we need to switch the cuda version during inference.
I Have been trying to run this on colab, i have 3 frames and running the video_extrapolation mode, i cant run this on a higher resolution ex: 960x720 because CUDA runs out of memory, when i downscaled for 640x480 it gave me a new error.
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 402, in video_completion_seamless
corrFlowF = calculate_flow(args, RAFT_model, video, 'forward')
File "video_completion.py", line 120, in calculate_flow
_, flow = model(image1, image2, iters=20, test_mode=True)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/raft.py", line 101, in forward
fmap1, fmap2 = self.fnet([image1, image2])
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/extractor.py", line 176, in forward
x = self.conv1(x)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 419, in forward
return self._conv_forward(input, self.weight)
File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 416, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[2, 4, 480, 640] to have 3 channels, but got 4 channels instead
Just wanted to ask if you are planning on simplifying the prerequisites by porting to only one version of pyTorch (tentative Nov.)? If that’s not an option maybe containerize the components to make them easier to use?
Impressive results btw. congratulations! And thanks for sharing your work!
can you export the pth for version1.4.0? I cant load the .pth file.
I get this error when running object removal on my own files :
Calculating backward flow 69 <---> 70
Finish flow prediction.
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 464, in video_completion_seamless
videoFlowF = complete_flow(args, corrFlowF, flow_mask, 'forward', FlowF_edge)
File "video_completion.py", line 194, in complete_flow
flow_mask_img = flow_mask[:, :, i] if mode == 'forward' else flow_mask[:, :, i + 1]
IndexError: index 16 is out of bounds for axis 2 with size 16
I install both python environments, and can successfully run the two projects.
Then I clone FGVC, and follows the instructions, it works well for the first step for raft with torch=1.6.0, but when I switch to FGVC (with torch=0.4.0), the torch is still raft, and run 1.6.0.
However, I have 0.4.0 in the pip list.
I tried to restart a new terminal, it still not work.
The python run with torch=1.6.0,I paste a screenshot below.
It seems that, no matter which python environments I use, the python script will change to torch=1.6.0
To solve #40, I compressed the images and reduced the quality. But in doing so, I am facing this error:
RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [3225, 1, 43, 75] and grid with sizes [3150, 9, 9, 2]
am I asking too much?
I'm trying to execute a sequence of 228 fullhd jpg files on Colab.
I get this message:
tcmalloc: large alloc 5673369600 bytes == 0x55b4a2bdc000 @ 0x7fa98570bb6b 0x7fa98572b379 0x7fa922e4892e 0x7fa922e4a946 0x7fa95b226bb3 0x7fa95acdb17a 0x7fa95acdc293 0x7fa95b1dcedf 0x7fa95ad6a6a6 0x7fa95ad6b7b0 0x7fa95afc21c9 0x7fa95a963689 0x7fa95b0a1919 0x7fa95ad6d09b 0x7fa95b157a08 0x7fa95a963689 0x7fa95b0a1709 0x7fa95ad62d1d 0x7fa95b161dc8 0x7fa95a963689 0x7fa95b0a1b29 0x7fa95cddb347 0x7fa95a963689 0x7fa95b0a1b29 0x7fa96a73b746 0x55b2ce0770e4 0x55b2ce076de0 0x55b2ce0eb6f5 0x55b2ce07869a 0x55b2ce0e6a45 0x55b2ce07869a
bash: line 7: 1201 Killed python3 video_completion.py --mode object_removal --path ../kepescap/kepesc --path_mask ../kepescap/kepesc_mask --outroot ../result/kep_removal --seamless
Colab link is not working
After generating a sequence of images, gif and the mp4 file I'm getting the following error on Centos 7.
Any ideas?
Traceback (most recent call last):
File "video_completion.py", line 615, in
main(args)
File "video_completion.py", line 578, in main
video_completion_seamless(args)
File "video_completion.py", line 558, in video_completion_seamless
mask, video_comp = spatial_inpaint(deepfill, mask, video_comp)
File "/home/megatron/FGVC/tool/spatial_inpaint.py", line 12, in spatial_inpaint
img_res = deepfill.forward(video_comp[:, :, :, keyFrameInd] * 255., mask[:, :, keyFrameInd]) / 255.
File "/home/megatron/FGVC/tool/frame_inpaint.py", line 35, in forward
_, inpaint_res, _ = self.deepfill(image.to(self.device), mask.to(self.device), small_mask.to(self.device))
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/DeepFill.py", line 25, in forward
stage1_output, resized_mask = self.stage_1(stage1_input, mask)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/DeepFill.py", line 58, in forward
x = self.down(x)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/ops.py", line 116, in forward
return self.out(x)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/FGVC/models/DeepFill_Models/ops.py", line 52, in forward
x = self.conv(x)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
input = module(input)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, **kwargs)
File "/home/megatron/anaconda3/envs/FGVC/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_EXECUTION_FAILED
Thanks for sharing such a great job! As said in the paper, a homography warp, which is estimated using RANSAC on ORB feature matches, is used before calculating the optical flow. I'm wondering where the implementation of this operation locates since I can't find it in the code.
in your code, videoNonLocalFlowF and videoNonLocalFlowB is None?
get_flowNN_gradient(args, gradient_x_filled, gradient_y_filled, mask, mask_gradient, videoFlowF, videoFlowB, None, None)
RuntimeError: CUDA out of memory. Tried to allocate 3.52 GiB (GPU 0; 15.75 GiB total capacity; 11.38 GiB already allocated; 1.07 GiB free; 11.54 GiB reserved in total by PyTorch)
(Running on Colab Pro)
What is the minimum required specs to run the current version of FGVC ?
I am using some different images and getting this error on Colab. I would be grateful to receive some advice on this. Thanks!
just like this link
Hi, I ran into this error, I tried to work it out myself and I found the cause but no real solution yet, I'm willing to find a reliable one and share it, if interested. Below I wrote a report detailing what I found:
Error text:
File "e:/Work/FGVC/tool/video_completion.py", line 120, in calculate_flow
_, flow = model(image1, image2, iters=20, test_mode=True)
File "C:\Users\eduard\anaconda3\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "e:\Work\FGVC\RAFT\raft.py", line 127, in forward
corr = corr_fn(coords1) # index correlation volume
File "e:\Work\FGVC\RAFT\corr.py", line 51, in __call__
corr = bilinear_sampler(corr, coords_lvl)
File "e:\Work\FGVC\RAFT\utils\utils.py", line 66, in bilinear_sampler
img = F.grid_sample(img, grid, align_corners=True)
File "C:\Users\eduard\anaconda3\envs\torch\lib\site-packages\torch\nn\functional.py", line 3390, in grid_sample
return torch.grid_sampler(input, grid, mode_enum, padding_mode_enum, align_corners)
RuntimeError: grid_sampler(): expected grid and input to have same batch size, but got input with sizes [2850, 1, 38, 75] and grid with sizes [2775, 9, 9, 2]
Run into error while running a cloned code from https://github.com/vt-vl-lab/FGVC Version 1.0 with the following command
video_completion.py --mode object_removal --path E:/Work/video_segmentation/images_/ --path_mask E:/Work/video_segmentation/masks_/ --outroot ../result
On a set of 10 images/masks pairs, with the shapes [3, 300, 600]
The functiongrid_sampler(...) is used by bilinear_sampler(...) having the arguments corr and coords_lvl with different shapes [2850, 1, 38, 75] and [2775, 9, 9, 2]
corr = bilinear_sampler(corr, coords_lvl)
which breaks grid_sampler(...) because of the different dimmensions
In raft.py the initialize_flow() function computes the size of the grid as being (1, 37, 75) from image with shape (1, 300, 600) because H/8 = 37.5 and H//8 = 37
def initialize_flow(self, img):
""" Flow is represented as difference between two coordinate grids flow = coords1 - coords0"""
N, C, H, W = img.shape
coords0 = coords_grid(N, H//8, W//8).to(img.device)
coords1 = coords_grid(N, H//8, W//8).to(img.device)
# optical flow computed as difference: flow = coords1 - coords0
return coords0, coords1
This propgates to corr_fn() -> CorrBlock.call() that receive the coordinates as being : torch.Size([1, 2, 37, 75])
coords0, coords1 = self.initialize_flow(image1)
#...
corr = corr_fn(coords1) # index correlation volume
corr = CorrBlock.corr(fmap1, fmap2)
batch, h1, w1, dim, h2, w2 = corr.shape
corr = corr.reshape(batch*h1*w1, dim, h2, w2)
self.corr_pyramid.append(corr)
for i in range(self.num_levels-1):
corr = F.avg_pool2d(corr, 2, stride=2)
self.corr_pyramid.append(corr)
This is used to reshape centroid_lvl in method call(...) to [2775, 1, 1, 2]), which is then used in bilinear_sampler() with its shape being torch.Size([2775, 1, 1, 2]), which in turn gives the shape to coords_lvl
corr eventually takes its shape from fmaps, as detailed in ch.III
Fmaps are generated (in this case) using a BasicEncoder(nn.Module). Looking at the forward(self, x) method,
def forward(self, x):
# if input is list, combine batch dimension
is_list = isinstance(x, tuple) or isinstance(x, list)
if is_list:
batch_dim = x[0].shape[0]
x = torch.cat(x, dim=0)
x = self.conv1(x)
x = self.norm1(x)
x = self.relu1(x)
x = self.layer1(x)
x = self.layer2(x)
x = self.layer3(x)
x = self.conv2(x)
if self.training and self.dropout is not None:
x = self.dropout(x)
if is_list:
x = torch.split(x, [batch_dim, batch_dim], dim=0)
return x
It returns a feature map from an image, by passing it through different nn layers, with its shape at the end being exactly torch.Size([256, 38, 75]). This shape is propagates as described in chapter III.
In raft.py, method self.fnet([image1, image2]) return fmap1 and fmap2, with shapes torch.Size([1, 256, 38, 75])
with autocast(enabled=self.args.mixed_precision):
fmap1, fmap2 = self.fnet([image1, image2])
This propagates to CorrBlock.init() to corr object with shape
torch.Size([2850, 1, 38, 75]) where 2850=38*75
It is then appended to self.corr_pyramid, to finally be used in the call(), in bilinear_sampler() with its shape being torch.Size([2850, 1, 1, 2])
To match the same shape, either the small CNN must be modified, or the way the grid shape its defined in initialize_flow(), from:
coords0 = coords_grid(N, H//8, W//8).to(img.device)
to:
coords0 = coords_grid(N, np.round(H/8), np.round(W/8))).to(img.device)
However, I suspect this change should be made at other points in the implementation as well.
This method of dividing by 8 to match the output shape of the convolutions can raise multiple errors, and should better match exactly the output shape. In case of modifying the architecture of the CNN, this will also throw shape miss-match errors.
Like the title says; it doesn't create these folders after processing. I would love to hear what I missed! :)
Hi there,
I was able to run the test (tennis) clips, but when I try it out on my own material I get the following error. I've tried it with 4K and HD material and I keep getting the same result:
miniconda3/lib/python3.8/site-packages/torch/nn/functional.py:3384: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn("Default grid_sample and affine_grid behavior has changed "
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 578, in main
video_completion(args)
File "video_completion.py", line 368, in video_completion
mask_tofill, video_comp = spatial_inpaint(deepfill, mask_tofill, video_comp)
File "/lgNet/users/andy/Downloads/FGVC-master/tool/spatial_inpaint.py", line 12, in spatial_inpaint
img_res = deepfill.forward(video_comp[:, :, :, keyFrameInd] * 255., mask[:, :, keyFrameInd]) / 255.
File "/lgNet/users/andy/Downloads/FGVC-master/tool/frame_inpaint.py", line 35, in forward
_, inpaint_res, _ = self.deepfill(image.to(self.device), mask.to(self.device), small_mask.to(self.device))
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/lgNet/users/andy/Downloads/FGVC-master/models/DeepFill_Models/DeepFill.py", line 27, in forward
stage2_output, offset_flow = self.stage_2(stage2_input, small_mask)
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "/lgNet/users/andy/Downloads/FGVC-master/models/DeepFill_Models/DeepFill.py", line 78, in forward
attn_x, offset_flow = self.CAttn(attn_x, attn_x, mask=resized_mask)
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
result = self.forward(input, **kwargs)
File "/lgNet/users/andy/Downloads/FGVC-master/models/DeepFill_Models/ops.py", line 343, in forward
yi = F.softmax(yiscale, dim=1)
File "/lgNet/users/andy/flameTimewarpML/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 1498, in softmax
ret = input.softmax(dim)
RuntimeError: CUDA out of memory. Tried to allocate 3.91 GiB (GPU 0; 23.88 GiB total capacity; 16.52 GiB already allocated; 2.16 GiB free; 20.72 GiB reserved in total by PyTorch)
Trying
%%shell
cd FGVC
cd tool
python /content/FGVC/tool/video_completion.py \
--mode video_extrapolation \
--path /content/FGVC/data/tennis \
--outroot /content/FGVC/result/tennis_extrapolation \
--H_scale 2 \
--W_scale 2 \
#--seamless
in Google Colab returns:
Pytorch 1.6.0 for flow prediction
Finish Calculating flow.
tcmalloc: large alloc 1651507200 bytes == 0x117076000 @ 0x7fb727903001 0x7fb71bdac765 0x7fb71be10bb0 0x7fb71be12a4f 0x7fb71bea9048 0x50a7f5 0x50cfd6 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x50b053 0x634dd2 0x634e87 0x63863f 0x6391e1 0x4b0dc0 0x7fb7274feb97 0x5b26fa
Completing forward flow 0 <---> 1
tcmalloc: large alloc 1767358464 bytes == 0x7fb589f70000 @ 0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c1395753 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
tcmalloc: large alloc 1767358464 bytes == 0x7fb5209f4000 @ 0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c139576a 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
tcmalloc: large alloc 1767358464 bytes == 0x7fb4cb160000 @ 0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c1395753 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
tcmalloc: large alloc 1767358464 bytes == 0x7fb5346dc000 @ 0x7fb7279011e7 0x7fb6c1382f9a 0x7fb6c1395254 0x7fb6c139576a 0x7fb6c13932f2 0x7fb6c13930a9 0x7fb6c1382a48 0x50a7f5 0x50cfd6 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x507f24 0x509c50 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x509918 0x50a64d 0x50c1f4 0x507f24
Completing forward flow 1 <---> 2
Completing forward flow 2 <---> 3
Completing forward flow 3 <---> 4
Completing forward flow 4 <---> 5
Completing forward flow 5 <---> 6
Completing forward flow 6 <---> 7
Completing forward flow 7 <---> 8
Completing forward flow 8 <---> 9
Completing forward flow 9 <---> 10
Completing forward flow 10 <---> 11
Completing forward flow 11 <---> 12
Completing forward flow 12 <---> 13
Completing forward flow 13 <---> 14
Completing forward flow 14 <---> 15
Completing forward flow 15 <---> 16
Completing forward flow 16 <---> 17
Completing forward flow 17 <---> 18
Completing forward flow 18 <---> 19
Completing forward flow 19 <---> 20
Completing forward flow 20 <---> 21
Completing forward flow 21 <---> 22
Completing forward flow 22 <---> 23
Completing forward flow 23 <---> 24
Completing forward flow 24 <---> 25
Completing forward flow 25 <---> 26
Completing forward flow 26 <---> 27
Completing forward flow 27 <---> 28
Completing forward flow 28 <---> 29
Completing forward flow 29 <---> 30
Completing forward flow 30 <---> 31
Completing forward flow 31 <---> 32
Completing forward flow 32 <---> 33
Completing forward flow 33 <---> 34
Completing forward flow 34 <---> 35
Completing forward flow 35 <---> 36
Completing forward flow 36 <---> 37
Completing forward flow 37 <---> 38
Completing forward flow 38 <---> 39
Completing forward flow 39 <---> 40
Completing forward flow 40 <---> 41
Completing forward flow 41 <---> 42
Completing forward flow 42 <---> 43
Completing forward flow 43 <---> 44
Completing forward flow 44 <---> 45
Completing forward flow 45 <---> 46
Completing forward flow 46 <---> 47
Completing forward flow 47 <---> 48
Completing forward flow 48 <---> 49
Completing forward flow 49 <---> 50
Completing forward flow 50 <---> 51
Completing forward flow 51 <---> 52
Completing forward flow 52 <---> 53
Completing forward flow 53 <---> 54
Completing forward flow 54 <---> 55
Completing forward flow 55 <---> 56
Completing forward flow 56 <---> 57
Completing forward flow 57 <---> 58
Completing forward flow 58 <---> 59
Completing forward flow 59 <---> 60
Completing forward flow 60 <---> 61
Completing forward flow 61 <---> 62
Completing forward flow 62 <---> 63
Completing forward flow 63 <---> 64
Completing forward flow 64 <---> 65
Completing forward flow 65 <---> 66
Completing forward flow 66 <---> 67
Completing forward flow 67 <---> 68
Completing forward flow 68 <---> 69
Completing backward flow 0 <---> 1
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
<ipython-input-2-4abcfdc683a5> in <module>()
----> 1 get_ipython().run_cell_magic('shell', '', 'cd FGVC\ncd tool\npython /content/FGVC/tool/video_completion.py \\\n --mode video_extrapolation \\\n --path /content/FGVC/data/tennis \\\n --outroot /content/FGVC/result/tennis_extrapolation \\\n --H_scale 2 \\\n --W_scale 2 \\\n #--seamless')
2 frames
/usr/local/lib/python3.6/dist-packages/google/colab/_system_commands.py in check_returncode(self)
136 if self.returncode:
137 raise subprocess.CalledProcessError(
--> 138 returncode=self.returncode, cmd=self.args, output=self.output)
139
140 def _repr_pretty_(self, p, cycle): # pylint:disable=unused-argument
CalledProcessError: Command 'cd FGVC
cd tool
python /content/FGVC/tool/video_completion.py \
--mode video_extrapolation \
--path /content/FGVC/data/tennis \
--outroot /content/FGVC/result/tennis_extrapolation \
--H_scale 2 \
--W_scale 2 \
#--seamless' died with <Signals.SIGKILL: 9>.
Literally just keeps saying this:
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 402, in video_completion_seamless
corrFlowF = calculate_flow(args, RAFT_model, video, 'forward')
File "video_completion.py", line 120, in calculate_flow
_, flow = model(image1, image2, iters=20, test_mode=True)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/raft.py", line 101, in forward
fmap1, fmap2 = self.fnet([image1, image2])
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/content/FGVC/RAFT/extractor.py", line 176, in forward
x = self.conv1(x)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 399, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[2, 4, 72, 128] to have 3 channels, but got 4 channels instead
I used wget.exe version 1.21.1 (latest) on Windows.
PS C:\Users\marc-\Downloads> .\wget.exe https://filebox.ece.vt.edu/~chengao/FGVC/data.zip
--2021-08-16 20:39:46-- https://filebox.ece.vt.edu/~chengao/FGVC/data.zip
Resolving filebox.ece.vt.edu (filebox.ece.vt.edu)... 128.173.88.43
Connecting to filebox.ece.vt.edu (filebox.ece.vt.edu)|128.173.88.43|:443... failed: Unknown error.
Retrying.
Anyone had this issue?
I first tried with your example and everything executes fine but when it comes to retrieving the output I get this message:
unable to retrieve /result/tennis_removal/frame_seamless_comp_final/ from backend
Hi! Can I use it directly with a video file? Putting the corresponding mask video on another folder? Or i need single frames?
I tried with single frames but seems that ram finishes very fast. With colab pro is it usable? In general this application can manage folder with 2600 files?
Issue: Program still asks me to switch to Pytorch 0.4.0 after I've switched to FGVC environment for the 2nd execution which has torch 0.4.0 installed
I've setup the environments as asked by the referenced githubs. Then I've tried the object removal example from the quickstart:
python video_completion.py \
--mode object_removal \
--path ../data/tennis \
--path_mask ../data/tennis_mask \
--outroot ../result/tennis_removal \
--seamless
which also worked almost flawlessly in the raft environment. It created the result folder while also filling the subfolders with segmented images that look right (for example:
).
After changing the environment to FGVC & executing it again it still asks me to switch to pytorch 0.4.0 (-> change environment to FGVC) after executing it.
conda list torch indicates I've indeed installed the correct torch version being 0.4.0:
"conda list torch" produces:
(FGVC) D:\Weltherrschaft\AI\FGVC\FGVC-master\tool>conda list torch
# packages in environment at D:\Programme\Anaconda\Anaconda\envs\FGVC:
#
# Name Version Build Channel
torch 0.4.0 pypi_0 pypi
torchvision 0.2.1 pypi_0 pypi
I've also tried pytorch 0.4.1, same issue. Even though the github suggests Linux I've ran this on Windows 10. I'm assuming this cannot be the root of the issue because everything else seemingly worked well.
PS: For every German: Don't mind the "Weltherrschaft" ;-)
(Edit) Solution: zxyjfj's answer provides a step by step guide which, considering the reactions, seems to fix the issue for everyone.
I fixed it myself by locating the print in the source code & removing the try/catch around it so it would direct me to where the code fails.
Turns out it didn't have anything to do with the pytorch version.
They tried to import something that didn't exist in the directory & was no package and it wasn't even needed.
At another point they used an absolute path which obviously won't work unless your directories are called exactly like theirs.
Hi there,
Thanks for the tool, it's pretty awesome! I noticed that *.core
intermediary files in FGVC/tool/
do not get cleanup up. The network seems to produce one every so often and I realized that that was happening after my hard drive filled with 300ish GB of these files. Anyways, just letting you know.
Thanks,
Leo
python video_completion.py \
--mode video_extrapolation \ --path ../data/tennis \ --outroot ../result/tennis_extrapolation \ --H_scale 2 \ --W_scale 2 \ --seamless
Pytorch 1.6.0 for flow prediction
Finish Calculating flow.
Traceback (most recent call last):
File "video_completion.py", line 615, in
main(args)
File "video_completion.py", line 578, in main
video_completion_seamless(args)
File "video_completion.py", line 418, in video_completion_seamless
video, corrFlowF, corrFlowB, flow_mask, mask_dilated, start_point, end_point = extrapolation(args, video, corrFlowF, corrFlowB)
File "video_completion.py", line 165, in extrapolation
corrFlowF[H_start : H_start + imgH, W_start : W_start + imgW, :] = corrFlowF_ori
ValueError: could not broadcast input array from shape (512,960,2,0) into shape (512,960,2,69)
I saw 3 pre-trained models. But I don't know how to train with my own dataset.
Hi, thank you for your wonderful work!
I want to know how to obtain the mask file for a custom video?(like the mask files in ./data/tennis_mask)
Thankyou for the great code.
"Results in the object removal setting, however, are difficult to compare and evaluate due to the lack of ground truth content behind the masked object. For this reason, we introduce a further synthetic object mask inpainting task. Specifically, we take a collection of free-form object masks and randomly pair them with other videos, pretending there is an object occluding the scene"
Can you please explain it with concrete example how you are evaluating your code on object removal task. Are you using some other dataset for which ground truth is available. I wanted to compare results for my project and I am not able to get the evaluation(PSNR, SSIM, LPIPS) and training code. Can you point me to some public library for this, it would be very useful.
Hi all, sorry for the inconvenience.
Please check the latest ver1.0 code. You only need to set up one environment, i.e., Pytorch 1.6.0, CUDA 10.1. There is no need to switch between different environments.
Please let me know if you have any questions. Thanks.
I'm using Colab to run this.
Whenever I use the video completion, the video output is very skippy. A couple of frames will play nicely, but then the video skips to a couple frames back and then quickly skips back. I can tell the video is being extrapolated, but this skipping makes playback not smooth.
Traceback (most recent call last):
File "video_completion.py", line 613, in
main(args)
File "video_completion.py", line 576, in main
video_completion_seamless(args)
File "video_completion.py", line 510, in video_completion_seamless
None)
File "/content/FGVC/tool/get_flowNN_gradient.py", line 440, in get_flowNN_gradient
videoNonLocalFlowB[:, :, :, 0, indFrame],
TypeError: 'NoneType' object is not subscriptable
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.