Comments (8)
@murillo128 The last option has the advantage of maintaining parity with WebRTC. Only potential disadvantage is that this would probably lead to having to reset all simulcast streams on receipt of an FIR for one of them (the current Chrome behavior).
from webcodecs.
The API has changed a lot since this issue was filed.
Not support it, and make the app create N encoder instances, and make the app support it. Not very likely as currently there is no support for accessing the image raw data, so it will have draw the media stream in n different canvas, downscale it, create the capture stream for each one and pass each one to an instance of the decoder.
This is now much better. You can create N encoders, configure them each with a different target encoding size, and they will internally resize your input VideoFrames as needed. You now also have raw access to the image data, but that's not required for resizing to happen internally.
Support it as configuration parameter of an encoder. This would allow to have single input for the simulcast encoder, but would make it much difficult to provide a good api compatible with non-encoders.
We have no plan to support simulcast as a parameter, but we do plan to support parameters to configure SVC (dependent on codec and UA support). Tracked in #40.
Provide a helper/adapter that can be created with a webrtc encoding-like object parameter and will create internally n encoders, exposing each one for controlling each simulcast layer individually. This would allow to have a single input, and the simulcast adapter will internally downscale/forward it to each of the encoders individually.
We don't plan to make this part of the API, but such a helper could be implemented in javascript on top of the first option.
Hopefully this addresses the issue and breaks out our follow up work. Please re-open if more to discuss :)
from webcodecs.
IMHO, this is really more of an implementation than an API issue. For example, can the WebCodecs implementation recognize that there are N encoders utilizing the same input, and optimize performance?
In WebRTC it is also possible to create N encoders, but simulcast performs better without having to clone the input MediaStreamTrack N times, as well as behaving more logically on a congested network (e.g. simulcast streams do not converge in resolution and framerate, as may happen with N distinct encodings from the same MediaStreamTrack).
Congestion control behavior is out of scope for the WebCodecs API, so it needs to be handled by the application. This is definitely non-trivial, so in practice it is best solved within a library written by someone who understands congestion control. For the algorithm to perform properly, it is necessary to obtain loss/throughput estimates for each of the simulcast streams. This might be handled purely at the application level (e.g. if the application is running RTP/RTCP over WebTransport), or there might b some assistance provided by the transport (e.g. WebTransport-stats, which are in the process of being revised to reflect Http3Transport).
from webcodecs.
We don't plan to make this part of the API, but such a helper could be implemented in javascript on top of the first option.
IMHO, this is really more of an implementation than an API issue. For example, can the WebCodecs implementation recognize that there are N encoders utilizing the same input, and optimize performance?
@chcunningham how would you recommend doing this, considering #104 and #129 (which are not yet merged but help somewhat)? This is one of those rather important use-cases that we need to get right, because the performance implications of having authors doing it "manually" are very important (lots of copies, lots of encoding time wasted). It's also not
It seems like ref-counting semantics would maybe allow this? Consider the following situation:
- Three video encoders are created, with a set of resolution r1, r2, r3
- A frame needs to be encoded, it's cloned twice (there is now three frames, but they point to the same underlying data)
encode()
is called in the same task on the three encoders- The UA starts processing the encoding jobs, and notices that it's the same input frames going to three encoders (because queued from the same task and the UA is dispatching the actual decode job from a stable state (or microtask, I think both work here).
- Bonus point: the UA notices the three encoders are in fact the same codec with different resolution and optimizes further
from webcodecs.
hrm I probably should have posted this other issues instead of resurrecting this old issue.
from webcodecs.
Discussed on last editors call. My recollection
- having 3 encoder recognize a common input and/or do processing of that input in the same task seems technically possible, but really tough to implement.
- a better option may be to just make this behavior explicit using 1 encoder.
- for instance, we could add a simulcast key to the VideoEncoderConfig which accepts a sequence of other config objects describing the additional resolutions. The output callback may then be called with chunks of different resolutions in a rotating fashion.
- demand for simulcast may not be huge (so far none that I'm aware of). we anticipate that SVC will come to be more important/common, particularly with AV1.
from webcodecs.
triage note: marking 'extension', as latest proposal (if any action taken) is to simply add knobs the config dict.
from webcodecs.
demand for simulcast may not be huge (so far none that I'm aware of)
One major use case would be to support the creation of multi-bitrate streams for HLS and DASH. Allowing the creation of these streams client-side would reduce the need for transcoding server-side.
from webcodecs.
Related Issues (20)
- Support for AV1 switch frames HOT 5
- Fatal encoding/decoding error on Windows 10/11 HOT 2
- Why is isTypeSupported() promise-based? HOT 8
- Receiving Uncompressed Webcam Data without Browser Compression HOT 5
- After how many decode should the codec process the frames? HOT 10
- Clarify `reset()` behaviour when multiple things are being output HOT 3
- key-frame request handling when scalability mode is not L1T1 for encoder HOT 6
- Clarify the `Clone configuration` algorithm HOT 4
- Figure out what should happen to the unused bits in 10-bits and 12-bits pixel formats
- Rephrase non-normative uses of RFC2119 keywords HOT 1
- Web Audio API compatibility HOT 1
- Assign VideoFrame resource to [[resource reference]] in BufferSource constructor HOT 4
- VideoFrameBufferInit metadata field missing
- sourceWidthBytes from sampleWidth, not sampleHeight
- Vekil
- Sporadic build failures HOT 4
- Define scope for w3c candidate recommendation HOT 3
- Candidate Recommendation tracking issue
- VideoFrame copyTo() behavior with non-RGBA/RGBX/BGRA/BGRX formats HOT 1
- VideoPixelFormat enum values do not follow casing rule guidelines HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.