Comments (6)
- read the raw data of captured media data, such as YUV video data, or PCM audio data,etc. Then the raw data could be passed to wasm/js module.
It is not clear why it is expected that the "raw data" of a video is "YUV" (see https://stackoverflow.com/a/39048445; https://github.com/brion/yuv-canvas).
The audio data can be expressed as an AudioBuffer
using decodeAudioData()
- If can decode is true, attempt to decode the encoded audioData into linear PCM. In case of failure, set can decode to false.
or a Float32Array
using AudioWorkletNode
, see https://github.com/web-platform-tests/wpt/blob/d5be80a86d4f938250c075ac12414ad47516969c/webaudio/js/worklet-recorder.js (see WebAudio/web-audio-api#2011 relevant to options
at the code).
- It will be better the capture could be worked on worker.
What does "capture" mean in this context?
- The data passing is efficient without memory copy.
The term "efficient" is not clearly defined. "efficient" compared to precisely what approaches that are not "efficient"?
- hardware accelerate encoder/decoder could be used by wasm/ js module, which is a Plus.
Not sure what that means. Encoding and decoding any unknown audio and/or video codec by "wasm/js"?
from webcodecs.
I did the wrong action of closing the issue.
-
we need to do the vidoe processing ourselved based on WASM. So we might need the RGBA or YUV data of video from WEBRTC capture. We could get the raw data of audio through webaudio API.
-
Capture here means "WebRTC Capture audio/video". It means we would like to get the raw data from worker scope.
-
some CPU has some hardware accelerating encoder/decoder. It will be better to expose the capability to applicaiton layer.
from webcodecs.
You can currently use ImageCapture.grabFrame()
to get ImageBitmap
instances of the captured video frames then process the frames. That procedure is possible right now, without waiting for WebCodecs to specify anything.
from webcodecs.
The API returns ImageBitmap , which is not the raw data. It must be drawed to canvas and then get the imageData from Canvas API.
from webcodecs.
You can read the image data via the VideoTrackReader.readable
and implement your encoder as a TransformStream
from webcodecs.
Old bug! Fidel and Chrome folks have discussed this directly, but for other folks reading along:
What we need is :
- read the raw data of captured media data, such as YUV video data, or PCM audio data,etc. Then the raw data could be passed to wasm/js module.
Yes, originally offered via VideoTrackReader / AudioTrackReader, but this is now deprecated in favor of (as of #131) using MediaStreamTrackProcessor. Demo here.
- It will be better the capture could be worked on worker.
Using MediaStreamTrackProcessor, you will have the option to transfer the readable
stream to a worker.
- The data passing is efficient without memory copy.
Working with WASM unfortunately imposes some copies, as described in WICG/reducing-memory-copies#1
- hardware accelerate encoder/decoder could be used by wasm/ js module, which is a Plus.
At least in Chrome, we will prefer acceleration by default where available. PR #118 will add the ability to strictly require that hardware acceleration be used (which will fail to create the codec when acceleration is not available).
Please re-open if more to discuss.
from webcodecs.
Related Issues (20)
- Can't use AVC video codecs on mp4 1920x1080 resolution HOT 6
- Is https://www.w3.org/TR/webcodecs/#dom-audioencoder-encode complete enough? HOT 1
- Specify cross-agent-cluster structured cloning for various Web Codecs API objects HOT 3
- The encoder/decoder configure methods is not using a queue to check for configuration support HOT 3
- github.dev/?redirect=%2Famirhfarzaneh%2Fjackmckew.dev%2Fblob%2F3a07c5e66d2c5635e0a32780888be2c6fe5a7dcb%2Fplugins%2Flatex&requestId=f3e355d5-2a0d-4dc1-9ddc-5df9a5c2a0af
- decodeQueueSize/encodeQueueSize is not updated on the task where dequeue event is fired HOT 5
- value of optimizeForLatency in emitted decoder configuration HOT 1
- Rounding direction for `AudioData.duration` HOT 1
- What does `allocationSize` return if formats in the source and destination are null? HOT 2
- Chunk data definitions are incomplete (VP9, AV1)
- What's the best way to ensure 1-in 1-out decoding for h264 video? HOT 7
- `contentHint`: notes aren't normative and shouldn't describe normative behaviour HOT 2
- frame.copyTo error in mac-chromeV118 HOT 2
- Probably a silly question: can this encode irregularly spaced frames? HOT 1
- Explicit frame drop signal from VideoEncoder HOT 1
- Long term reference support for AVC/HEVC HOT 6
- isConfigSupported: definition of "invalid" HOT 4
- webcodecs encode h264/h265 Capability set HOT 1
- Support for AV1 switch frames HOT 5
- Fatal encoding/decoding error on Windows 10/11 HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from webcodecs.