fast rendering?

[bbc]

fast rendering?

[MatthewShotton]

hey @gossi
I've just pushed v0.51.1 which should help speed up the rendering while seeking.

The problem was the videonode was waiting for the underlying video elements readyState to be greater than the HAVE_FUTURE_DATA value before it started updating the texture rather than the HAVE_CURRENT_DATA value which would've been enough to render the current frame.

It might still not be quite as fast as a standard video element as there's a number of extra operations happening under the hood when a seek take place. But it should be a fair bit faster than it was previously.

[gossi]

I gave it a quick try. Here is my code:

ctx.video(getClipVideoElement(clip));

// ...

	getClipVideoElement(clip) {
		const elements = this.get('clipElements');
		if (!elements.has(clip.get('id'))) {
			const element = document.createElement("video");
			element.setAttribute("crossorigin", "anonymous");
			element.setAttribute("webkit-playsinline", "");
			element.setAttribute("playsinline", "");
			element.src = getClipUrl(clip);
			this.loadClip(element, clip);
			elements.set(clip.get('id'), element);
		}

		return elements.get(clip.get('id'));
	}

	loadClip(element, clip) {
		fetch(getClipUrl(clip)).then(response => {
			if (response.ok) {
				return response.blob();
			}
		}).then(blob => {
			element.src = URL.createObjectURL(blob);
		});
	}

Unfortunately, this doesn't help, scrolling is as laggy as before. It takes a realizable amount of time until the blob is ready and put onto the video element. I tried to workaround by setting element.src to the clip url at first, unfortunately didn't help. It was also slower to do ctx.video(elem) instead of ctx.video(url) (maybe my subjective perception).

As it is no difference for the video element to take a url or stream, maybe there are some interaction artifacts between the video elements, the canvas or the animation frame?

[MatthewShotton]

hey @gossi

I expect this might be to do with the encoding of the videos you are using. If the video is encoded with any P or B frames and you seek to a point between I-frames the underlying video element has to decode a number of frames before it is able to display the correct frame. This can introduce a lag, and can vary a fair amount between different browsers and video codecs.

One potential way around this is to encode your videos as I-Frame only which should help speed up the seeking. I've just done some tests with two videos, where one was i-frame only and noticed a significant speed up in the seeking.

[gossi]

hey @MatthewShotton

interessting. I never had that kind of frames for video encoding on my agenda regarding this. Since I didn't know about different frames, I was happy to learn about, I found this article which provides very understandable illustrations. However, I don't think this affects the problem here. I used a video, which I also used before switchting to VideoContext that worked fine with scrolling.

I have another idea but it's only a suggestion by me, since I don't know the relevant code part of VideoContext. Before I should explain my setup a little more. My own timeline is moved with a mouseMove listener. A listener that basically fires "everytime" unless the mouse is stopped. Before switching to VideoContext I used plain <video> elements and set the currentTime similar to now. On each move event, the video frame for that timeframe is requested to be painted on the video element, which of course resulted in a lot of requests but also made it kinda smooth. That said a video with 12MBps bitrate was scrolled smoothly (once it was loaded, I didn't know the memory trick from above that time).
With VideoContext the updates happen on a tick (= requestAnimationFrame). The question here is, are the video frames are also requested on each tick and then painted (passed from the video element to the canvas) or are they requested each time currentTime is set but only requested to paint on each tick? Maybe this is a reason for not-immediately-smooth-video-seeking-when-playhead-is-changed-on-mousemove.

If so, what might be good to have in such a situation is to kinda "deactivate" the animation frame or ignore it when setting currentTime. Moving a cursor is mostly delimited by a mousedown and mouseup event, one could do:

mousedown: VideoContext.setInstantRendering(true)
mousemove: VideoContext.currentTime = newTime
mouseup: VideoContext.setInstantRendering(false)

might something like this work? Is it a good idea at all?

[MatthewShotton]

hey @gossi
Thanks, that's interesting. If it worked when you were just using a video element it should work with the VideoContext, so it looks like it is most probably a bug. I'll have a dig into this today or tomorrow and let you know progress.

[gossi]

Wow @MatthewShotton you are awesome!

Seeking now is up to the same speed as with <video> before. So I thought I give it test. Here is what I found:

1. Memory isn't faster
   I tried using the load-the-video-into-memory-first-hack from jsconf again, however I could see no difference. I should be trying to use this on rare <video> element first to see if this makes a difference I guess and if, how and if this could be applied onto VideoContext - I skipped that part.

2. Initial loading
   Give the most simple example, that one video is connected to the VideoContext taking the whole timeline of VideoContext, on initial loading it takes unexpectedly long until the first frame is painted. This is something where the <video> element is faster at the moment. It kinda displays the first frame immediately. It feels like, the VideoContext is waiting for the whole video loaded until it will display the first frame. Maybe also something like the readyState thing as before?
   (Since I'm working with local videos, the videos are loaded instantly for me and because of that a delay immediately attracts attention and results in negative perception)