Comments (25)
CtxSyn code not ready yet? I'm really looking forward to it.
from sepconv-slomo.
My apologies for not not been able to publish our reference implementation yet, and thank you for being interested in our work.
- The methodology described in the training section only applies to the synthesis network, we trained the optical flow in advance and treat it as an off-the-shelf component.
- It sounds like you are changing the aspect ratio when resizing the training data. I would recommend not to change the aspect ratio and to select training data through cropping instead. Note that the flow as well as the synthesis model are fully-convolutional, which is why we do not need to resize the input during inference.
from sepconv-slomo.
Look forward to the code of cvpr 2018 paper.
from sepconv-slomo.
Thank you for your feedback. 🙂
It might take a while but we are planning to eventually release it just like we did with SepConv, stay tuned!
from sepconv-slomo.
this is one of my favorite AI, I was wondering how well it would perform with 3D renderer image, could this AI be used to reduce the number of frame to render and create intermediary frame for free ? would it also be possible to use the motion data generated by the renderer to produce even better interpolated frame using the AI ? could we also even use the depth or even ID map ?
I'm a software developer and I'd really like to get started into AI, how difficult would it be to use your new context aware AI ?
sorry for all the question, i'm very excited about the idea to halve the amount of time to render 3D animation !
from sepconv-slomo.
Thank you for your interest in our work.
- Yes, it is possible to render only a subset of the total frames and interpolate the missing ones, thus potentially obtaining the rendered video more quickly.
- Yes, if the renderer provides per-pixel motion data then the synthesis network can leverage it more effectively than the optical flow estimation, effectively improving the interpolation results.
- Yes, the synthesis network can be augmented with depth as well as semantic maps, allowing it to leverage this additional information to potentially improve the interpolation results.
Our newest paper is rather elaborate and requires custom layers, I would thus recommend waiting until we release our implementation.
from sepconv-slomo.
I am afraid that I have not received the approval to do so yet. I am, like you, very interested in getting the code out there as soon as possible. After all, I would like to see our work be put to good use.
from sepconv-slomo.
I guess that your new method and a new video card will be viable (and probably awesome) way of viewing/converting personal daily movies.
Do you think that your methods or something similar could also be used to estimate pixels in x/y (super-resolution) instead of estimating pixels in time (between frames) ?
Finally in a other subject are you aware of any interesting 3d reconstruction papers/projects using continuous image sequences ( videos of a single scene ) ?
Thanks for sharing your awesome researches.
from sepconv-slomo.
Single-image super resolution is an interesting topic that is orthogonal to our work. However, video super resolution and video frame interpolation are related in that they both need to perform motion compensation. One could thus try to apply ideas from one area to the other.
Scene reconstruction, like single-image super resolution, is an interesting topic that is orthogonal to our work. Fortunately, once could make use COLMAP which is a magnificent open-source project. The assumption of a continuous image sequence is commonly made when finding corresponding points between images.
from sepconv-slomo.
It’s great work!, I also looking forward to your code. But waiting is not my favorite, I'm trying to implement your work on my own
I have some questions when I've training this model.
Could you answer me some quenstions? it would be great pleasure if you answer.
-
Should flow generating network(like pwc-net in your paper) be pre-trained? When I trained your model from scratch it didnt work well but when i used pretrained model. It seems to starting work.
-
I'm using open dataset (UCF-101) and resize to small size( 384 x 384). trained model works well(I think) when input size is smaller than 384x384 but larger than this size interpolated output seems not good. It seems predicting flow is failed and the output is blurred even if theres no movement. Should I rescale the output of the flow generating net? when I test different resolution inputs?
Thanks for reading I'm not native speaker in English so sentences could be weird.
from sepconv-slomo.
@sniklaus Thanks for your fast reply.
-
Oh, that was my misunderstanding. I thought only context extractor network(vgg or resnet) needed to be trained and the estimating bidirectional flow is just used in front of synthesis net. (after training flownet, generating frame works better, but network generates frame before training flownet though)
-
Thanks for your advice. My question was not good.
- This is an example image to describe my problem
when I test trained model using this image for example. network generates star between stars in frame 0.5(256x256), but It fails when I used high resolution frames. It generates star like frame 0.5(512x512)
I don't understand this(I think result should be same).
Thanks for your fast reply again. and have a nice weekend. thanks
from sepconv-slomo.
@sniklaus,
thanks again for your answer,
your paper is very interesting, and I have one question about
spatial warping block : is it the same warping block as PWC warping bloc ? Thanks your very much in advance for your answer .
from sepconv-slomo.
@dlwtojd26 If you double the resolution then everything will be twice as far away in terms of pixels. My guess is that your network cannot handle the increased optical flow and hence produces the artifacts that you are experiencing.
@sampiet Thank you for being interest in our work. We perform forward-warping for the spatial warping block whereas PWC-Net relies on backward-warping.
from sepconv-slomo.
@sniklaus
Thank you for your reply. In fact, I implemented the entire network using Tensorflow, so that's why PWC-Net block is based on (https://github.com/daigo0927/PWC-Net_tf) . So, I used the same PWC-Net warping block for your spatial warping (is it make sense for your ?). But the problem is that I have a lot of artifact in my images out of the network. I think the problem comes from this spatial warping block. So my question how to correct it based on your previous comment? Second, how to manage occlusions in the spatial warping block? Thank you in advance, sorry to take a lot of your time.
from sepconv-slomo.
@sniklaus thanks for reply. I will check the implementation. I’m using this implementation.(https://github.com/daigo0927/PWC-Net_tf)
from sepconv-slomo.
@sniklaus
Great works!
I'm sorry, I am trying to reproduce your work according to your paper, but I have a question about that how do you implement the spatial warping function which has three inputs(context map, image, optical flow) and two outputs(image, context map).Could you tell me what the warping function works inside?
Thank you!
from sepconv-slomo.
@sniklaus It's a great work about video frame interpolation! I'm trying to implement it by myself. While I am a little confused about some details of your spatial warping.
- How to pad the patch when warping, zeros or other methods?
- If the holes after warping need to be filled?
- Could you provide more explanation about the process "measure the flow quality by checking the brightness constancy and discard contributions from flow vectors that significantly violate this constraint."?
I'm sorry if I missed something in the paper. Thank you for reading patiently. Have a good weekends!
from sepconv-slomo.
Any news about the release date of your implementation ?
from sepconv-slomo.
My apologies for still not being able to release the reference implementation of CtxSyn. Please note that I am eager to do so, but have unfortunately not gotten the approval yet.
from sepconv-slomo.
hello sniklaus, the implementation of ctxsyn is ok now?
from sepconv-slomo.
It’s great work!, I also looking forward to your code. But waiting is not my favorite, I'm trying to implement your work on my own
@dlwtojd26 ,hello dlwtojd26,can you share your implement of the cyxsyn work? it would be great,thanks !
from sepconv-slomo.
My apologies for still not being able to release the reference implementation of CtxSyn. Please note that I am eager to do so, but have unfortunately not gotten the approval yet.
Is it possible for you to give a more detailed description or reference here about the spatial warping in your paper? It still confused me for how to perform such a warping strategy even after reading your paper for many times.
from sepconv-slomo.
For everyone interested in the forward warping, please consider taking a look at our recently released softmax splatting: https://github.com/sniklaus/softmax-splatting
from sepconv-slomo.
from sepconv-slomo.
We just released the full inference code to our CVPR 2020 paper on softmax splatting which is the direct successor to our CVPR 2018 paper on the context-aware synthesis here: https://github.com/sniklaus/softmax-splatting
I am hence closing this issue for now, thanks everyone for your patience and my apologies you had to wait this long!
from sepconv-slomo.
Related Issues (20)
- see_you_again_540.mp4 HOT 1
- vaule of interpolation HOT 1
- Make it work with pytorch installations from wheel packages HOT 1
- YUV HOT 1
- resolution HOT 1
- test HOT 1
- nn.Upsampling is deprecated. Use nn.functional.interpolate instead. HOT 1
- Flow calculation HOT 3
- Perceptual loss HOT 42
- some question about softmax-splatting HOT 1
- some question about code HOT 1
- Need your help to analyze the reason HOT 8
- Assertion Error in Video Frame interpolation HOT 1
- How to train a model HOT 1
- a question about the loss function LF HOT 1
- A question about the Instance normalization HOT 1
- load_state_dict_from_url() got an unexpected keyword argument 'file_name' HOT 1
- How to understand the meaning of the intIndex in sepconv.py? HOT 1
- AttributeError: 'Tensor' object has no attribute 'clip' HOT 2
- Checkpoint Downloading HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sepconv-slomo.