Cache a node: GL.Node saveBuffer+cached props

[gre]

Cache a node: GL.Node saveBuffer+cached props

[morukutsu]

Hi, I was interested by the "previous frame feature" described in the first post (to implement motion blur / accumulation like effects). Using Surface.captureFrame was too slow for real time as expected.

I implemented a quick demo of the feature here: morukutsu/gl-react-dom@4b6705e

There are still some issues:

• implementation is hacky because of the magic number I use for the previous frame fbo id
• previousFrame is a hardcoded name which triggers the feature, using the "GL.PreviousBuffer" semantic would be better
• I didn't check the implementation for the native version
• The previous frame works at the Surface level. Ideally, it would be nice to be able to retrieve the previous frame of any GL Node.

What are your plans regarding this feature? I guess it is not really useful for most use cases but I kinda like it! And it's not too hard to implement properly.

[gre]

Sorry to have been a bit too talkative in my 5-days-ago comment, but basically most issues I was talking about are fixed in a PoC implementation I've started this weekend ( see also https://github.com/ProjectSeptemberInc/gl-react/issues/70#issuecomment-254172991 ). Still a WIP, I'm exploring different things and still change the API a lot, so I won't publish this right now, but soon I hope ;) I will also have to see how I can port this to native, and how I can reuse the maximum code on JS side (which also might change some API).

Basically I do what you said: less magic in the lib and more control to user, in short: dropping the dedupe logic, embracing React lifecycle.

I'm using the context to connect things together.

One "hack" is that GL.Node render() method is like: <span>{all uniform sub nodes}</span>, so it's rendered normally in React (and i can use lifecycle!! important part). I also even put that inside the <canvas> tag (it can have content, it's just not visible. well, perfect!!)

I attach one framebuffer to each GL.Node instance. (except for the "back buffering" usecase, you just set a backbuffering prop, and you have 2 fbos that get swaped each draw time)

I have not yet thought about sharing and reusing fbos, attaching one fbo per GL.Node makes things so much simpler and allow to do things like caching (redraw only if changes, otherwise keep the fbo in memory) as well as allowing to snapshot a specific GL.Node without need to redraw it (just takes what's in the fbo). But still, I know there might be some usecases where you want to make fbos more global (i'm thinking about blur with lot of passes OR in native case, when you have lot of concurrent surfaces). I was considering a framebufferGroup concept, where you would set a number to make a sharable fbo across many nodes... anyway, that's complexifying the implementation, so i'll see later.

[gre]

<GL.Node saveBuffer="foo"> will be a good use-case to allow the use of discard gre/gl-react-native-v2#19

[gre]

maybe worth introducing a new decorator GL.Cache instead of adding more into GL.Node. The thing is maybe we would want to factorize some of this mecanism for contents too ( #53 & #56 ).

<GL.Cache shouldUpdate={false} key="foo">
   ...
</GL.Cache>

[alexpyzhianov]

@morukutsu I'm really interested in this feature as well, your solution seems to work, do you have any plans to refine it and make a PR?

[morukutsu]

@SereznoKot hi, yep I have some plans.
I used my workaround in my app and I had some issues: it's only possible to get the last frame of the whole surface, not per GL.Node. In my project I have a complex stack of effects and I would like to have local feedback effects.

So I'm thinking about something in the lines of what @gre suggested:

<GL.Node shouldUpdate={false} saveBuffer="foo"> ... // to retrieve the texture as a uniform GL.fromBuffer("foo")

I went through the source code some weeks ago and it doesn't seem too hard to implement. The current problem is that I'm not too familiar with the native part (gl-react-native, which has both Android and iOS low level implementations). So if I start working on it it would be for the gl-react-dom first.

[alexpyzhianov]

@morukutsu cool! Let me know, if you need any help with that (not sure, if I can help with native, more likely with dom)

[gre]

thanks for your interest guys, it motivates me more for moving gl-react to a better implementation :)

as side notes to this discussion, I'm not very satisfied with how gl-react is implemented today to easily move forward to this more complex step (the fact gl-react handle the tree of VDOM itself, unfold the tree, etc.. it's quite complex code already). I think gl-react basically needs to embrace more implementing things in the React way. I'm considering reimplementing it inspiring from how react-art and react-music is implemented.

I can't guarantee when this will come but I want to make a quick PoC to see if it's feasible, and figure out what tradeoffs will be made:

• basic idea is truely benefiting React diffing, so we have less code in gl-react, probably better perf, and for sure cache will be easier to implement. Imagine if a GLNode is a host component (dom/native) that hold some gl states like the Framebuffer. We could easily just introduce a snapshot method that directly hit that framebuffer and gets its pixels.
  Also, might just need implementshouldComponentUpdate to having caching? :)
  Imagine if you can also have state in part of your stack of effects!
• the advanced feature of sharing VDOM to dedup the shared rendering will probably be removed and instead, user will have to define what is shared, maybe a bit like in react-music bus idea. React knows how to handle tree of VDOM, but you can't represent graphs in that paradigm. It's an advanced use-case anyway.

// idea
<Surface>
    <HelloTexture>
        <GL.Shared from="foo" />
    </HelloTexture>
    <GL.Shared to="foo">
        <Blur>{...}</Blur>
    </GL.Shared>
</Surface>

lot of possibilities, but I need to see how to make this possible, as a GL.Node is not something on top of DOM or UIView but it's a shadow abstract thing for working with GL, and the GLSurface is the true renderer at the end. One subtle but hard problem is how to handle the rasterized content? It's an edge case too, i'm wondering why not just using React ref that you would have to provide to a uniform texture. Only problem in that is it means rendering again, because at render time, you don't have the ref. need some kind of indirection maybe? (like a lazy function, called in DidUpdate lifecycle)

thoughts?

[morukutsu]

I had a look at the VDOM dedupe feature. It's very useful but as you said, it is a very complicated piece of code. Also, I'm wondering if most of the use cases does not really takes advantage of this feature.
If so, it make sense to remove this code in favor of a simpler implementation and let the user manage his own caching mechanism.

We could easily just introduce a snapshot method that directly hit that framebuffer and gets its pixels.

If there is an active OpenGL context during the GL.Node render (which is not the case at the moment). The GL.Node can be rendered to a FBO. Then, this FBO is passed to the parent (another GL.Node or a Surface for final display). The problem is that we need one surface sized FBO for each GL.Node in the tree if caching is enabled. If not, old FBOs can be reused.

Basically, the Surface rendering code is moved to the GL.Node class. Surface would just manage the OpenGL context and pass it to the children.

One subtle but hard problem is how to handle the rasterized content?

I'm not sure I understand everything about this problem. React ref are a nice mechanism, however, assuming rasterized content is rendered to a FBO. Using a helper to retrieve FBOs from a global state is not bad either (the GL.fromBuffer("foo") thing ).

[morukutsu]

Oh great, then I can't wait to see what you will come up with.
I agree, the easier & clean way is not to reuse FBOs. I like the framebufferGroup idea though!
In most use cases, it shouldn't use too much VRAM ;)

[gre]

This is now solved in gl-react v3 ! It's a complete rewrite.

• see release article: http://greweb.me/2016/12/gl-react-v3/
• see cookbook/api: https://gl-react-cookbook.surge.sh
• gl-react v3 is still in development. (hosted on: http://github.com/gre/gl-react )
• gl-react v2 is still the stable version (hosted on: https://github.com/ProjectSeptemberInc/gl-react )

Relevant example that show this issue is fixed: https://gl-react-cookbook.surge.sh/golrot and many others