Specify options or a method to encode pixel dimensions (width and height) of individual frames

[guest271314]

Specify options or a method to encode pixel dimensions (width and height) of individual frames

[guest271314]

After various tests had concluded that the issue with Chromium browser not displaying variable pixel dimension encoded WebM files output by MediaRecorder was HTML <video> element implementation, as this code https://github.com/guest271314/MediaFragmentRecorder/blob/imagecapture-audiocontext-readablestream-writablestream/MediaFragmentRecorder.html outputs pixel dimensions of the initial width and height, and is not resized at Chromium, though is resized at Mozilla Firefox.

However, this code https://plnkr.co/edit/4JxS4O?p=preview which uses a slightly modified version of https://github.com/thenickdude/webm-writer-js although the frame rate is not correct displays variable width and height and dispatches resize event.

Developers should have the ability to pass options to the encoder which either encode a single initial width and height and does not resize, or output variable width and height and resizes.

[pthatcherg]

WebCodecs (as currently designed) does not do containerization. It does encode/decode and the JS/wasm can choose how to containerize. Thus, you don't need to work around limitations/bugs in various browsers related to container formats because the JS/wasm would be responsible for doing so.

For example, you could use WebCodecs to encode VP8 and then use your own JS to create the WebM container with whatever settings you want in the container. WebCodecs settings would only be for things related to the encoding, not the containerization.

[guest271314]

@pthatcherg This issue does not address "containerization". This issue is intended to address settings passed to the codec encoder.

[pthatcherg]

Sorry if I misunderstood. Could you perhaps give an example of what you think the parameters would look like? We have definitely have a mechanism for setting parameters (see here).

It sounds like you want one for setting the dimensions. But what if the dimensions of the input frame don't match the parameters? Do you want the codec to scale before encoding?

[guest271314]

Re

WebCodecs (as currently designed) does not do containerization

What is "muxer" intended to mean at the explainer, other than combining audio, video, subtitles, etc. into a single output?

But what if the dimensions of the input frame don't match the parameters?

Not sure what you mean. The concept is to control the entire process.

[pthatcherg]

The "muxer" in the example is meant to be JS code, not an API from the browser. In fact, "Direct APIs for media containers (muxers/demuxers)" is listed as a non-goal.

If you create a video encoder configured to 720p resolution and feed it a frame that's 1080p, would do you expect it to do?

[guest271314]

It depends on the requirement. One use case might be to scale all frames to 720, which Chromium <video> does at output. Another use case might be to read and encode each frame to the input image pixel dimensions. Expect this API to do exactly what the initial and/or dynamic configuration settings state.

[pthatcherg]

It sounds like what you're asking for is a "scaleBy" parameter which defaults to 1 (no scaling). There's a similar parameter in WebRTC that allows scaling before encoding.

[guest271314]

Am asking that all parameters that the encoder used be capable of being set both initially and dynamically. Note also that contraints that are specified for getUserMedia() do not apply to MediaStreamTracks, for example from canvas.captureStream() or HTMLMediaElement.captureStream(). Though the fact might not be immediately clear, each individual specification which uses MediaStreamTracks (e.g., ImageCapture, canvas.captureStream(), getDisplayMedia(), HTMLMediaElement.captureStream()) need to define their own constraints, both what is expected and what is not expected. This is clear at Firefox, where HTMLMediaElement.captureStream().getVideoTracks()[0].getSettings() returns an empty JavaScript plain object {}, while getSettings() on a MediaStreamTrack from getUserMedia() returns the constraints of the track, e.g., see https://bugzilla.mozilla.org/show_bug.cgi?id=1537986.

[pthatcherg]

Oh.....

Yes, setting parameters dynamically will be possible. Doing that properly is what #5 is about.

[guest271314]

Yes, that could resolve part of the issue. Since WritableStream gets data from ReadableStream, both parts of the transform stream need to have the capability to dynamically change the setting during the read/write.

[pthatcherg]

Which part of the issue is not resolved by dynamically changing the settings?

[guest271314]

@pthatcherg

WebCodecs (as currently designed) does not do containerization. It does encode/decode

Does WebCodecs intend to write and use its own video decoder? Or rely on an existing decoder at the implementation?

[chcunningham]

The spec offers a mechanism to configure (and reconfigure) encoder sizes.
https://wicg.github.io/web-codecs/#dictdef-videoencoderconfig

The intent is that an implementing UA must actually produce frames at the configured size.

Feel free to re-open if more to discuss.