Comments (14)
The correct way to handle the incoming video would be to modify the StreamRecorder
, namely the void StreamRecorder::OnFrame(const cricket::VideoFrame& yuvframe)
method and pass the video to OpenCV instead of handling with FFmpeg.
from libsourcey.
Were you able to find a solution? Please close this if fixed.
from libsourcey.
@auscaster yes that makes sense, thanks!
from libsourcey.
@auscaster I'm revisiting this; do you have an example of passing the video to OpenCV in the OnFrame
method you mention above? I'm guessing there's some conversion of a yuvframe
to a Mat
, etc, and wanted to check if there was an example of this in the codebase. Thanks!
from libsourcey.
I am interested in this as well, it would be nice to have some example.
from libsourcey.
I think an example of this would make a nice contribution to the repo, so I've posted a bounty to help get this done.
from libsourcey.
Hey @fohara nice work on the bounty. So to be clear you're just looking for a bi-directional webrtc cricket::VideoFrame
to cv::Mat
converter? Not looking to use OpenCV capture sources or anything like that?
from libsourcey.
Please check this pull request for sample app:
#206
BTW: Didn’t detect that there is a 100 bucks bounty for this :) @fohara take it for free, just a Christmas gift ;)
from libsourcey.
@auscaster using an OpenCV capture source sounds reasonable, but I'm still getting familiar with these APIs. Ultimately, whatever fits the use case best, which is to:
- Analyze a live video stream frame-by-frame with OpenCV (which I assume entails a
cv::Mat
conversion) - Record the video stream, in addition to analyzing it
- Optionally stream a modified video back to client, i.e. video that has bounding box annotations, or has been processed with a canny edge detector
An example that accomplishes the above points, along with a README and self-contained project structure similar to the other WebRTC samples, is what I'm looking for.
Let me know if that makes sense, thanks.
from libsourcey.
@lutsykigor nice! It looks like you're delivering on some of the points I mentioned in my previous comment. I'll take a look this evening (PST) and see if I can get it running. Thanks!
from libsourcey.
@fohara the easiest way for you to run a sample client is to use Firefox(it should work out of the box), I have just added extra comments to my pull request a few seconds ago regarding running it on a Chrome, please check updated Readme.md
file. I do not have access to my work PC right now, so I have just edited these files online :-)
Chrome does not allow you to access media capabilities on HTTP host, you need to be on HTTPS. You will need to make simple tweaks to change sample Node.js web server to serve requests on HTTPS.
I hope it helps. Feel free to ask any questions if you have problems running this sample.
According to your points in previous post:
Analyze a live video stream frame-by-frame with OpenCV (which I assume entails a cv::Mat conversion)
You do not need to run heavy neural network computations on each frame, as the situation won't change every frame. Please note that in my sample I am using single shot detector plus mobile nets which is very light weight, if you plan to use some heavy neural networks you must have few CUDAs to process analytics realtime fore every frame (I am just guessing here).
Record the video stream, in addition to analyzing it
You can use webrtcrecorder sample for this
Optionally stream a modified video back to client, i.e. video that has bounding box annotations, or has been processed with a canny edge detector
I do not think that you want that :-) To draw some analytics data on server side, you need to decode every single frame, run analytics on it (you may skip analyzing some frames actually), draw analytics output on every frame and encode it before sending back to clients. Thus you will have huge delays in video delivery and a substantial server resources exhaust. Check my comments in the pull request description. You can just decode frame, run neural network computations only on some frames, and send analytics data back to web clients by a websocket connection, then you can use JavaScript do draw it on a client side.
Regards
from libsourcey.
@fohara I've merged the OpenCV sample created by @lutsykigor into master. I look forward to hearing how it goes for you!
from libsourcey.
@lutsykigor @auscaster ok I was able to get libsourcey
to compile with the opencvanalyzer
sample app, but I got a few errors when I attempted to run it. My eventual goal is to run this within a Docker container, so I started putting a Dockerfile together to build an image with all the dependencies and configurations required to run opencvanalyzer
.
In the meantime, this Dockerfile should be useful for debugging the errors I encountered since it will provide a consistent build environment. With that in mind, here are my repro steps:
- Download the work-in-progress Dockerfile I have posted in this gist
- Build the Dockerfile with (this will take a while):
docker build -t libsourcey-webrtc-opencvanalyzer . | tee docker_build.log
- Start an interactive bash session in the Docker container with:
docker run -i -t libsourcey-webrtc-opencvanalyzer /bin/bash
- Finally, attempt to run the sample help with:
./webrtc/samples/opencvanalyzer/opencvanalyzer -h
That last command prints out a bunch of config errors:
[000:000] [17] (webrtcvoiceengine.cc:974): webrtc: failed to connect context, error=-1
[000:000] [17] (audio_device_pulse_linux.cc:171): failed to initialize PulseAudio
[000:003] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:005] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: unable to open playback device: No such file or directory (-2)
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:006] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: unable to open record device: No such file or directory
[000:007] [17] (audio_device_generic.cc:51): BuiltInAECIsAvailable: Not supported on this platform
[000:007] [17] (audio_device_generic.cc:61): BuiltInAGCIsAvailable: Not supported on this platform
[000:007] [17] (audio_device_generic.cc:71): BuiltInNSIsAvailable: Not supported on this platform
[000:007] [17] (adm_helpers.cc:47): Unable to set recording channel to kChannelBoth.
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:007] [17] (adm_helpers.cc:56): Unable to access microphone.
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:007] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_inputMixerHandle, ) error: No such file or directory
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: unable to open record device: No such file or directory
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:008] [17] (adm_helpers.cc:100): Unable to access speaker.
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib control.c:954:(snd_ctl_open_noupdate) Invalid CTL
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: snd_mixer_attach(_outputMixerHandle, ) error: No such file or directory
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: GetDevicesInfo - Could not find device name or numbers
ALSA lib pcm.c:2266:(snd_pcm_open_noupdate) Unknown PCM
[000:008] [17] (webrtcvoiceengine.cc:974): webrtc: unable to open playback device: No such file or directory (-2)
terminate called after throwing an instance of 'std::runtime_error'
what(): Cannot connect to Symple server. Did you start the demo app and the Symple server is running on port 4500?
Aborted
These errors indicate that I've likely missed some configuration steps, which should be evident by what's in the Dockerfile. Once these errors are resolved, I can proceed to installing the node.js client app, but I wanted to make sure this part was solid first. Thanks!
from libsourcey.
@fohara great job on a containerization, I am just scratching a surface here but this is what I found:
- You are running an
opencvanalyzer
binarry inside a container only, but you need to run singlaling(symple server - last error highlights that) and web server as well, please refer to sample readme file, these both servers should be started with single Node.js app. - All these exceptions from WebRTC stack, probably, are thrown because of a lack of actual playback and capture devices inside a container. I do not know much about running Google’s WebRTC native stack inside a container, maybe this will help: https://github.com/DamonOehlman/webrtc-testbot
- Seems that you are installing develop version of OpenCV, from the master branch. I do not recommend doing that, try to use tagged
3.3
version instead.
In general, I suggest you to run sample app without containeriazation. It can help you to troubleshoot future problems related to inserting that app to a container.
I hope that my answers will help you, but probably there are still a lot of other issues to resolve.
from libsourcey.
Related Issues (20)
- New ffmpeg version HOT 3
- Academic research on vulnerable c++ code snippet
- Academic research on vulnerable c++ code snippet HOT 2
- Unable to build WebRtcRecoder on Ubuntu 16.04.6 LTS HOT 2
- STUN Implementation
- Failed to build libsourcey on FreeBSD in library av
- iOS build with webrtc on mac
- Error building libSourcey with WebRTC (Errors in AddRef and Release)
- Assertations of thread ids
- Wrong frame rate of mp4 file encoded from local camera.
- How to build for mobile?
- crashed when webrtcstreamer have a new connection
- Documentation pages are broken HOT 2
- multiplexencoder.cpp updateStreamPts() Invalid pts for both streams HOT 2
- Trying to convert Opencv Mat to webrtc::I420Buffer HOT 1
- Can anyone tell me how to use webrtccapturer sample ?
- LibSourcey does not support modern version of WebRTC, because cricket::VideoCapturer was removed from WebRTC HOT 2
- libwebrtc_full.a undefined symbol HOT 1
- Fail compilation on default setting. (fedora33) HOT 1
- Error generating Visual Studio 16 2019
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libsourcey.