Comments (7)
Why does WebCodecs need to do anything? In the same implementation, is it likely that the encoder for a B-frames codec (say h264) would generate a bitstream that the decoder wouldn't be able to handle?
I suppose there are some implementations with software encoders and hardware decoders. Is that what you were thinking?
from webcodecs.
It impacts two things:
-
The API of the decoder. If B-frames are handled internally, I believe you need to pass a timestamp into the decoder letting it know the order things should come out in. I don't believe that info is in the bitstream, but I could be wrong. Also, you need to know how many frames will be buffered to know how much internal delay there is (how many frames you need to give it before frames come out the decoded side).
-
The API of the encoder. You need to control if you want B-frames or if you want a low-latency encoder, and perhaps how far the the B-frames can reach (impacting the amount of buffering required by the decoder).
-
Capabilities for the encoder and decoder related to 1 and 2 above.
from webcodecs.
I used to implement and develop H.264/AVC software encoder at my former job.
So please let me comment from its perspective.
(I'm sorry that my comment doesn't take care of other codec at all)
- The API of the decoder. If B-frames are handled internally, I believe you need to pass a timestamp into the decoder letting it know the order things should come out in. I don't believe that info is in the bitstream, but I could be wrong.
A decoder doesn't need a timestamp for reordering frames which includes B-frames. Because each coded frame has Picture Order Count
which represents output order in each slice header. So it is ok for decoder just to be pushed the Access Unit in decoding order, then you can get each decoded frame in output order.
Additionally, encoder and decoder basically don't treat a duration of each frame. (You can describe frame rate in VUI, though.) While to play correctly needs timestamp and duration. I think a container should take care of it. So decoder/encoder API should be able to pass through this information with target frame.
Also, you need to know how many frames will be buffered to know how much internal delay there is (how many frames you need to give it before frames come out the decoded side).
I'm not sure about decoder implementation in detail. But I think decoder could output immediately a frame if it is output order. Anyway, container should take care of timing for decoding and presenting.
- The API of the encoder. You need to control if you want B-frames or if you want a low-latency encoder, and perhaps how far the the B-frames can reach (impacting the amount of buffering required by the decoder).
I think it depends on the encoder capability. Encoder will decide which frame type should be used in the end. (e.g. IDR:IDR, I:I, P:I/P, B:I/P/B)
So if user want a low-latency stream, then they set just IPPP sequence, I think.
And picture decoded buffer size in decoder is specified by Profile and Level.
from webcodecs.
The main limitation we have here is that not all platform decoders have the option to output frames in decode order. The common denominator then is that WebCodecs video decoders should output frames in presentation order. Some implementations would need to buffer out-of-order frames to make that happen.
from webcodecs.
I think that in case we support b-frames in both encoder/decoders, the frames should be output in encoding/decoding order and have a TransformStream that reorders them
12345->Encoder-> I(1)P(5)B(2)B(3)B(4)->Decoder->15234->transformstream->12345->mediastream
We should add pictId
to the VideoFrame
so the reordering is possible.
from webcodecs.
I think that it is difficult to take a presentation number from decoding order frames. It will not be able to find a number of frames between reference frames...
So a decoder WebCodecs uses should take care of it by using information in raw stream.
Or if a decoder can label a pictId
to the VideoFrame
, it'll be good, too.
A limitation size of buffering out-of-order frames must be specified by each codec specification.
from webcodecs.
As others have pointed out, #55 is basically a duplicate of this, but with more current insights and more recent feedback. Closing this one in favor of that.
The API of the encoder. You need to control if you want B-frames or if you want a low-latency encoder, and perhaps how far the the B-frames can reach (impacting the amount of buffering required by the decoder).
This is tracked in greater detail by #57
from webcodecs.
Related Issues (20)
- Receiving Uncompressed Webcam Data without Browser Compression HOT 5
- After how many decode should the codec process the frames? HOT 10
- Clarify `reset()` behaviour when multiple things are being output HOT 3
- key-frame request handling when scalability mode is not L1T1 for encoder HOT 6
- Clarify the `Clone configuration` algorithm HOT 4
- Figure out what should happen to the unused bits in 10-bits and 12-bits pixel formats
- Rephrase non-normative uses of RFC2119 keywords HOT 1
- Web Audio API compatibility HOT 1
- Assign VideoFrame resource to [[resource reference]] in BufferSource constructor HOT 4
- VideoFrameBufferInit metadata field missing
- sourceWidthBytes from sampleWidth, not sampleHeight
- Vekil
- Sporadic build failures HOT 4
- Define scope for w3c candidate recommendation HOT 3
- Candidate Recommendation tracking issue
- VideoFrame copyTo() behavior with non-RGBA/RGBX/BGRA/BGRX formats HOT 1
- VideoPixelFormat enum values do not follow casing rule guidelines HOT 2
- EncodedAudioChunkInit should probably also support AllowSharedBufferSource HOT 1
- Issue
- numberOfChannels/sampleRate check in AudioDecoderConfig/AudioEncoderConfig HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.