maartenbreddels / ipywebrtc Goto Github PK

View Code? Open in Web Editor NEW

233.0 11.0 39.0 11.1 MB

WebRTC for Jupyter notebook/lab

License: MIT License

Python 34.20% JavaScript 54.27% TypeScript 11.53%

jupyter-notebook jupyter webrtc webcam video-streaming python ipython ipython-notebook

ipywebrtc's Introduction

ipywebrtc

WebRTC and MediaStream API exposed in the Jupyter notebook/lab.

See this tweet for a demo screencast.

Why use ipywebrtc?

Using ipywebrtc you can create a MediaStream out of:

Any ipywidget.
A video file.
An image file.
An audio file.
Your webcam/camera.

From this MediaStream you can:

Record a movie.
Record an image snapshot.
Record an audio fragment.
Stream it to peers using the simple chat function
Use it as a texture in ipyvolume

Demos

WebRTC and ipyvolume

Use remote MediaStreams and show them in 3d using ipyvolume.

ImageRecorder

Record and image from any stream for postprocessing.

WidgetStream

Turn any widget into a MediaStream.

Installation

To install:

$ pip install ipywebrtc                             # will auto enable for notebook >= 5.3

For a development installation (requires npm),

$ git clone https://github.com/maartenbreddels/ipywebrtc
$ cd ipywebrtc
$ pip install -e .
$ jupyter nbextension install --py --symlink --sys-prefix ipywebrtc
$ jupyter nbextension enable --py --sys-prefix ipywebrtc
$ jupyter labextension develop . --overwrite

ipywebrtc's People

Contributors

Stargazers

Watchers

ipywebrtc's Issues

Is from_file loading the entire file into memory?

It seem like the from_file routine is trying to load the entire file into memory, is this true?

Many Thanks
Max

Binder demo issues

The Widgetstream.ipynb requires requirements.txt dependencies pythreejs and ipyleaflet.

The ipyleaflet extension also throws an error/cannot be instantiated.

Unreliable capture of pythreejs with ImageRecorder on Safari

I am trying to capture an image of a pythreejs renderer using ImageRecorder. This works reliably on Chrome but not on Safari. I seem to be hitting a weird race condition, but I do not understand the asynchronous Javascript code involved well enough to figure what is going wrong.

Here is an example demonstrating the problem:

from pythreejs import *
ball = Mesh(geometry=SphereGeometry(),
            material=MeshLambertMaterial(color='red'))
light    = DirectionalLight(color='white', position=[3, 5, 1])
c        = PerspectiveCamera(position=[0, 5, 5], children=[light])
scene    = Scene(children=[ball, c])
renderer = Renderer(camera=c, scene=scene, controls=[OrbitControls(controlling=c)])
display(renderer)
#-----------------------------------------------
import ipywebrtc
stream = ipywebrtc.WidgetStream(widget=renderer, max_fps=30)

rec = ipywebrtc.ImageRecorder(filename='capture', format='png', stream=stream)
rec.autosave = True
#-----------------------------------------------
rec.recording = True
#-----------------------------------------------
rec.recording
# Usually still prints `True` :(

(Comments delimit notebook cells.)

I expect rec.recording = True to (a) rerender the scene, (b) capture and save an image, and (c) reset rec.recording to False. This all happens reliably on Chrome, but (b) and (c) almost always fail on Safari (no image is written, and rec.recording is still True at the end). Manually triggering a second redraw (e.g., by orbiting the view) usually causes (b) and (c) to finally happen.

As best I can tell, ImageRecorder's snapshot function hangs on the onCanPlay call. It appears the redraw requested by pythreejs here is somehow happening "too soon," but I don't understand what order everything needs to happen in.

This is tricky to debug since the capture starts working more (but not 100%) reliably when I insert breakpoints, e.g. here and here.

Do you have any ideas what might be wrong or suggestions on how to debug this?

Error displaying widgt:model not found

Uncaught（in promise）Error： Module jupyter-webrtc，manager.js：319 semver range ~0.5.0 is not registered as a widget module ad WidgetManager.loadClass (manager.js:319) at WidgetManager. at step at Object.next

I get above error in my jupyter lab width code:
from ipywebrtc import VideoStream
video = VideoStream.from_url("local mp3 file")
video

my soft version is:
jupyter code : 4.6.3
jupyter-notobook:6.0.3
ipython:7.13.0
ipykernel:5.2.1
jupyter client:6.1.3
jupyter lab:2.1.0
nbconvert:5.6.1
ipywidgets:7.5.1
ipywebrtc:0.5.0
jupyterlab-server:1.1.1

please help me! thank you

Control size (height x width) of video playback.

When I playback a video, it is very large and I have to scroll around to see all of it. Is it possible to control the size of the video using the widget? I tried setting height and width in the layout but that just changed the size of the window one could watch the video through (I hope that makes sense) not the size of the underlying video.

container for images/gifs

can I do real-time processing on the web audio stream?

Hi I noticed there are examples doing real time image processing from CameraStream. Is there a way to do a similar real time processing with AudioRecorder()? I would like to try some real time audio transcription based on it.

With Context for MediaStream Classes

Creating MediaStream classes using a with context can be useful as it ensures that media devices will be closed at the end of context.

Example syntax:

constraints = {
    "facing_mode": "user",
    "audio": False,
    "video": {
        "width": 640,
        "height": 480
    }
}

with CameraStream(constraints=constraints) as camera:
    recorder = ImageRecorder(stream=camera)
    ...

cannot import name 'Video' from 'ipywidgets'

Getting error when I am typing this syntex

import numpy as np
import ipyvolume as ipv

Error:

ImportError Traceback (most recent call last)
in
1 import numpy as np
----> 2 import ipyvolume as ipv

~\Desktop\My_Work\3_Freezer usability\ipyvolume-master\ipyvolume-master\ipyvolume_init_.py in
6 from ipyvolume import datasets # noqa: F401
7 from ipyvolume import embed # noqa: F401
----> 8 from ipyvolume.widgets import * # noqa: F401, F403
9 from ipyvolume.transferfunction import * # noqa: F401, F403
10 from ipyvolume.pylab import * # noqa: F401, F403

~\Desktop\My_Work\3_Freezer usability\ipyvolume-master\ipyvolume-master\ipyvolume\widgets.py in
10 import numpy as np
11 import ipywidgets as widgets # we should not have widgets under two names
---> 12 import ipywebrtc
13 import pythreejs
14 import traitlets

~\anaconda3\lib\site-packages\ipywebrtc_init_.py in
3 import ipywidgets as widgets
4 from ._version import version_info, version
----> 5 from .webrtc import *
6
7

~\anaconda3\lib\site-packages\ipywebrtc\webrtc.py in
11 Undefined
12 )
---> 13 from ipywidgets import DOMWidget, Image, Video, Audio, register, widget_serialization
14 from ipython_genutils.py3compat import string_types
15 import ipywebrtc._version

ImportError: cannot import name 'Video' from 'ipywidgets' (C:\Users\itadmin\anaconda3\lib\site-packages\ipywidgets_init_.py)

import ipyvolume as ipv
import numpy as np

ImportError Traceback (most recent call last)
in
----> 1 import ipyvolume as ipv
2 import numpy as np

~\anaconda3\lib\site-packages\ipywebrtc_init_.py in
3 import ipywidgets as widgets
4 from ._version import version_info, version
----> 5 from .webrtc import *
6
7

ImportError: cannot import name 'Video' from 'ipywidgets' (C:\Users\itadmin\anaconda3\lib\site-packages\ipywidgets_init_.py)

Extract frame from the camera as a np array into class self attribute

I want to extract the frame from the camera as a np array to use in other areas of my class. I tried making on_value_changed a self.methood to update frame but it didn't work and i don't know how to return the frame.

Any idea how i can get it all to be in one class?

class get_footage():
    frame = np.array(None)
    out = Image()
    
    def on_value_changed(_):
        im = Soruce.open(io.BytesIO(image_recorder.image.value))
        frame = np.array(im)
        return frame
    
    def get_frame(self):
        camera = CameraStream.facing_user(audio=False)
        image_recorder = ImageRecorder(stream=camera)
        print(image_recorder.image.observe(get_footage.on_value_changed,'value'))
        image_recorder.recording = True

JupyterLab 2.x

When will be compatible with JupyterLab 2.x?
Thank you!

ipywebrtc not working for jupyterlab 1.2.3

I am running jupyter lab 1.2.3 with python3
I installed the extension that way:

jupyter labextension install jupyter-webrtc
pip install ipywebrtc

When I run it in jupyterlab, the widgets don't render:

did I miss an install step?

(it is working on the jupyter notebook)

jupyter-webrtc doesn't play well with jupyterlab watch mode

I always get this issue with jupyter-webrtc (but I believe only the second time in run jupyter lab watch!)

$ jupyter labextension list                                                                                                                                                                                                       working
JupyterLab v0.33.0rc1
Known labextensions:
   app dir: /Users/maartenbreddels/miniconda3/envs/working/share/jupyter/lab
ipyvolume
        ipyvolume v0.5.2-dev.1  enabled  OK*
jupyter-matplotlib
        jupyter-matplotlib v0.3.0  enabled  OK
jupyter-threejs
        jupyter-threejs v1.0.0  enabled  OK
jupyter-webrtc
        jupyter-webrtc v0.4.1  enabled  OK*

   local extensions:
        ipyvolume: /Users/maartenbreddels/src/ipyvolume/js
        jupyter-webrtc: /Users/maartenbreddels/src/ipywebrtc/js

   linked packages:
        jupyter-webrtc: /Users/maartenbreddels/src/ipywebrtc/js

$ jupyter lab watch 

...........

> [email protected] prepare /Users/maartenbreddels/src/ipywebrtc/js
> npm run build:lib


> [email protected] build:lib /Users/maartenbreddels/src/ipywebrtc/js
> tsc --project .

jupyter-webrtc-0.4.3.tgz
Traceback (most recent call last):
  File "/Users/maartenbreddels/miniconda3/envs/working/bin/jupyter-lab", line 11, in <module>
    sys.exit(main())
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "<decorator-gen-7>", line 2, in initialize
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/notebook/notebookapp.py", line 1605, in initialize
    self.init_server_extensions()
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyterlab/labapp.py", line 213, in init_server_extensions
    load_jupyter_server_extension(self)
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyterlab/extension.py", line 145, in load_jupyter_server_extension
    watch(app_dir, logger)
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyterlab/commands.py", line 217, in watch
    return handler.watch()
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyterlab/commands.py", line 466, in watch
    self._populate_staging()
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyterlab/commands.py", line 920, in _populate_staging
    'linked_packages')
  File "/Users/maartenbreddels/miniconda3/envs/working/lib/python3.6/site-packages/jupyterlab/commands.py", line 1034, in _update_local
    os.remove(pjoin(dname, existing))
FileNotFoundError: [Errno 2] No such file or directory: '/Users/maartenbreddels/miniconda3/envs/working/share/jupyter/lab/staging/linked_packages/jupyter-webrtc-0.4.1-1146c25a4b8e99ba8bfe0daab26652cead4f710f.tgz'

Using external webcam

How do I create a CameraStream object for an external webcam connected to my system? The computer has a front and rear facing camera, but I need to use a specialized camera that shows up as another webcam on the computer. I am not sure how to create the constraints to use this camera.

Handle case when browser does not have access to camera or microphone

At the moment, this results in the the view not being properly created. Maybe we could degrade to show a meaningful error message.

Put this github repo in the documentation somewhere

Please make it easy for people to find the docs ... I googled around and eventually found it. Your pypi page listed the homepage as jupyter.org which isn't helpful.

JupyterLab 0.33.4: Module jupyter-webrtc, semver range ~0.3.0 is not registered as a widget module

Upon running CameraStream notebook with ipywidgets 7.3.1 and jupyterlab 0.33.4 I'm getting the following errors in the Chrome console:
@jupyter-widgets/base/lib/manager-base.js:273 Could not instantiate widget
at utils.js:119 Error: Could not create a model.
@jupyterlab/services/lib/kernel/default.js:928 Exception opening new comm
@jupyterlab/services/lib/kernel/default.js:129 Module jupyter-webrtc, semver range ~0.3.0 is not registered as a widget module

JupyterLab 0.33.4: Error: Cannot find module 'jupyter-webrtc/lib/labplugin'

In JLab 0.33.4 I'm getting the error(s) in the Chrome console:
Error: Cannot find module 'jupyter-webrtc/lib/labplugin'
Error: Cannot find module 'jupyterlab-volume/'
though both are installed.

Using ipywebrtc using raw byte datastream

Hi, I have an application where I want to have video streaming and snapshot taking ability, but the source of the data is an external camera with a C++ sdk (which I converted to Python using ctypes) rather than a webcam. Assuming I know image size and the data is 16-bit unsigned integer, how would I use ipywebrtc to stream this raw data?

I like the ability to have a videostream in the background (for example, in an output view in JupyterLab) while processing snapshots of the video stream in the notebook itself. My core issue is that the data is being streamed from a URL through proprietary frame grabbing code to a numpy array. I'm not sure if there is a way to turn this into a streaming object in the background that can be processed in later cells based on snapshots taken when those cells are run?

image.value is empty byte string

I'm having a strange issue. When I capture an image from the image recorder, it's value ends up being the empty byte string. If I then ask for the value in the subsequent cell, I get the bytes. What's causing this asynchrony? I tried to "fix" it by adding a delay between defining image and asking for its value, but that didn't work. What's the "correct" way of reliably grabbing the current camera frame's value?

blank image when saving WidgetStream as PNG

I followed the Example with ipyleaflet: streaming of a map widget to save an ipyleaflet map as a PNG image. However, the result is a blank black image. The ImageRecorder documentation seems to have the same issue. Any advice? @maartenbreddels @martinRenou

from ipyleaflet import Map
m = Map(center=(46, 14), zoom=5)
m

widget_stream = WidgetStream(widget=m, max_fps=1)
widget_stream

from ipywebrtc import WidgetStream, ImageRecorder
image_recorder = ImageRecorder(stream=widget_stream)
display(image_recorder)

Module jupyter-webrtc, semver range ~0.5.0 is not registered as a widget module

Hello,
I have an issue while running the template notebook in my environment.
Below is the web browser error log

(anonymous) @ manager-base.js:274
9523.cf94b68f6edde7eefd61.js:1 Uncaught (in promise) Error: Module jupyter-webrtc, semver range ~0.5.0 is not registered as a widget module
    at x.loadClass (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
    at x.<anonymous> (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:608)
    at step (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:389)
    at Object.next (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:370)
    at static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:364
    at new Promise (<anonymous>)
    at __awaiter (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:360)
    at x.ManagerBase._make_model (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:602)
    at x.<anonymous> (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:591)
    at step (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:389)
523.cf94b68f6edde7eefd61.js:1 Error: widget model not found
    at x.get_model (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
    at u.renderModel (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
renderModel @ 523.cf94b68f6edde7eefd61.js:1
manager-base.js:274 Could not instantiate widget
(anonymous) @ manager-base.js:274
utils.js:119 Error: Could not create a model.
    at promiseRejection (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:174)
    at async _handleCommOpen (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
    at async KernelConnection._handleCommOpen (static/lab/jlab_core.cc2967caa8ed93f04335.js?v=cc2967caa8ed93f04335:72622)
    at async KernelConnection._handleMessage (static/lab/jlab_core.cc2967caa8ed93f04335.js?v=cc2967caa8ed93f04335:72790)
promiseRejection @ utils.js:119
default.js:1044 Exception opening new comm
_handleCommOpen @ default.js:1044
2523.cf94b68f6edde7eefd61.js:1 Uncaught (in promise) Error: Module jupyter-webrtc, semver range ~0.6.0 is not registered as a widget module
    at x.loadClass (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
    at x.<anonymous> (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:608)
    at step (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:389)
    at Object.next (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:370)
    at static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:364
    at new Promise (<anonymous>)
    at __awaiter (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:360)
    at x.ManagerBase._make_model (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:602)
    at x.<anonymous> (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:591)
    at step (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:389)
manager-base.js:274 Could not instantiate widget
(anonymous) @ manager-base.js:274
utils.js:119 Error: Could not create a model.
    at promiseRejection (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:174)
    at async _handleCommOpen (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
    at async KernelConnection._handleCommOpen (static/lab/jlab_core.cc2967caa8ed93f04335.js?v=cc2967caa8ed93f04335:72622)
    at async KernelConnection._handleMessage (static/lab/jlab_core.cc2967caa8ed93f04335.js?v=cc2967caa8ed93f04335:72790)
promiseRejection @ utils.js:119
default.js:1044 Exception opening new comm
_handleCommOpen @ default.js:1044
2523.cf94b68f6edde7eefd61.js:1 Uncaught (in promise) Error: Module jupyter-webrtc, semver range ~0.6.0 is not registered as a widget module
    at x.loadClass (lab/extensions/@jupyter-widgets/jupyterlab-manager/static/523.cf94b68f6edde7eefd61.js:1)
    at x.<anonymous> (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:608)
    at step (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:389)
    at Object.next (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:370)
    at static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:364
    at new Promise (<anonymous>)
    at __awaiter (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:360)
    at x.ManagerBase._make_model (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:602)
    at x.<anonymous> (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:591)
    at step (static/lab/2713.ac2507cee52cdf02d439.js?v=ac2507cee52cdf02d439:389)

Here is my Jupyter labextension info:

jupyter labextension list
JupyterLab v3.2.5
/opt/conda/share/jupyter/labextensions
        jupyter-webrtc v0.6.0 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)

Other labextensions (built into JupyterLab)
   app dir: /opt/conda/share/jupyter/lab

I installed ipywebrtc following conda comment in the doc

Please help me fix this issue

new release of jupyter-sphinx

This project lists jupyter-sphinx in its sphinx configuration or requirements file, but does not appear to use this extension directly anymore.
We have just rewritten jupyter-sphinx from scratch, and fully changed the interface.
Therefore we would like to give you a heads up, in case you find the new version useful.

Now all code execution is done via a single jupyter-execute sphinx directive, instead of the previous ipywidgets-setup/ipywidgets-embed.
The new version is also much more flexible: it allows to embed any output recognized by jupyter in sphinx documentation. See the project documentation for details.

We have released 0.2.0rc1 on pip, please give it a shot using pip install 0.2.0rc1 --pre — we would love to hear your feedback.
Please also let us know by responding here or in jupyter/jupyter-sphinx#33 if the new release disrupts your workflow.

Use ImageRecorder on Jupyter_Video_Widget videos

Hi,

I want to use the ImageRecorder widget on a video stream of a custom widget. I use Jupyter_Video_Widget to be able to jump to special locations in a video. Under to hood it serves the video through a local webserver under http://127.0.0.1:57478/.
I thought about using the WidgetStream and the ImageRecorder to get the currently shown image in the video, but only a white image is shown.
Do you have an idea why it is not working, or another solution to my problem?
Thanks a lot!

Here is a working example showing my issue:
https://github.com/seidlr/jupyter-video-screenshot

Add streaming capability

It could be interesting to add real audio / video streaming capability.

I am not sure how that would work in detail, but this: https://github.com/jlaine/aiortc could be a reusable implementation of the webrtc protocol in Python.

The idea would be that you could generate sound in the backend (e.g. DSP filters or similar stuff) and then stream the result to the browser / frontend.

Stream camera from serverside /dev/video

What would be the correct way to stream a camera from a /dev/video device connected to the Jupyter server and not the remote clients web interface? Is there a direct way or does it need an additional streaming server?

Dev install from clean environment does not work

From a clean environment containing JupyterLab 1.0.0 and ipywidgets 7.5.0, installing ipywebrtc master does not seem to work. It fails with the error:

ws is not defined

In a file containing only:

module.exports = ws;

This seems to be triggered by the require('mqtt') which does a require('ws')

Using undefined reject

The reject function on this line seems undefined. What is it meant to do?

can I do real-time editing/filtering on the web stream?

I was wondering if I can apply some filters/mask on the streaming video? This requires access to the frames in real-time. Is it possible to do this on this package?

I have tried to capture frame one by one using cv2: clean the previous one and display the next one. However, what I actually see is firstly the first image disappears and the output space of this cell also disappears; then the next image comes and increases the output space. Overall, you will the whole page is shaking up and down due to the clean and display step.

It would be great if I can do the modification on the streaming on this package! Thanks.

TraitError: The 'filename' trait of an ImageRecorder instance must be a unicode string

Hello,

I would like to apply visual processing on stream of my camera but...

from ipywebrtc import CameraStream, ImageRecorder
camera = CameraStream(constraints=
                                      {'facing_mode': 'user',
                                       'audio': False,
                                       'video': { 'width': 640, 'height': 480 }
                                       })
camera
image_recorder = ImageRecorder(stream=camera)
image_recorder

I get an error by IPython on my jupyter lab :
TraitError: The 'filename' trait of an ImageRecorder instance must be a unicode string, but a value of traitlets.Undefined <class 'traitlets.utils.sentinel.Sentinel'> was specified.

Best,

Extension install breaks ipywidgets

I have this working when jupyterlab-webrtc is disabled

from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
def f(x):
    return x
interact(f, x=10);

But when I enable it (and refresh the page), the code above will no longer output a widget but instead complain with Error displaying widget.

!jupyter labextension enable jupyterlab-webrtc

The browser console logs for running the code above from a cell with the jupyterlab-webrtc extension enabled:

Setup details

nbextensions

labextensions

jupyter lab --version

0.33.11

jupyter notebook --version

5.6.0

Generalization of Camera Stream in context of IP Cameras / Headless Jetson Nano

I have been able to install ipywebrtc on a headless Jetson Nano with multiple IP Cameras (aka remote-cameras). I am accessing the ipywebrtc through jupyter notebook on another machine (aka local-machine).

The network is dynamic (DHCP). Let's assume that the addressing the IP cameras and authenticating to IP cameras is a solved issue (auto-discovery through nmap, auth information stored elsewhere)

In other words,
cap = cv2.VideoCapture("rtsp://user:[email protected]:554/cam/realmonitor?channel=1&subtype=1")
ret, frame = cap.read()
returns a good frame that can be displayed through plt.imshow()

The CameraStream makes use of the specific camera installed on the local-machine. How to enable the use of remote cameras through the CameraStream?

Video plays in notebook but not in JupyterLab

When I run an example similar to what is in the ImageRecorder demo as a notebook, the video plays automatically. For example a cell containing:

video = VideoStream.from_url( "http://localhost:8000/video.mp4") video

In a Jupyter notebook runs as expected, showing the video. However in Lab it just shows the class string as output:

VideoStream(video=Video(value=b'http://localhost:8000/video.mp4"))

Is there some magic command that needs to run or some setting in Lab? I know Lab disables javascript to some degree, perhaps that has something to do with it?

Problems In Linux Installation

I wasn't able to use ipywebrtc for Linux. Is it supported? I tried both the conda installation and building from source. Building from source gives the following error:

Obtaining file:///home/ipywebrtc Complete output from command python setup.py egg_info: setup.py entered $PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin running egg_info running jsdeps 6.9.0 6.9.0 Installing build dependencies with npm. This may take a while... npm WARN prepublish-on-install As of npm@5, prepublishscripts are deprecated. npm WARN prepublish-on-install Usepreparefor build steps andprepublishOnlyfor upload-only. npm WARN prepublish-on-install See the deprecation note innpm help scriptsfor more information. npm WARN lifecycle [email protected]~prepublish: cannot run in wd [email protected] webpack (wd=/home/ipywebrtc/js) npm WARN lifecycle [email protected]~prepare: cannot run in wd [email protected] npm run build:lib (wd=/home/ipywebrtc/js) npm WARN [email protected] requires a peer of ajv@^6.0.0 but none is installed. You must install peer dependencies yourself. npm WARN [email protected] No license field. npm WARN optional SKIPPING OPTIONAL DEPENDENCY: [email protected] (node_modules/fsevents): npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for [email protected]: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
audited 7154 packages in 5.29s found 26 vulnerabilities (2 low, 19 moderate, 5 high) runnpm audit fixto fix them, ornpm audit for details rebuilding js and css failed missing files: ['/home/ipywebrtc/ipywebrtc/static/extension.js', '/home/ipywebrtc/ipywebrtc/static/index.js'] Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/ipywebrtc/setup.py", line 177, in <module> setup(**setup_args) File "/opt/anaconda3/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup return distutils.core.setup(**attrs) File "/opt/anaconda3/lib/python3.7/distutils/core.py", line 148, in setup dist.run_commands() File "/opt/anaconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/opt/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/ipywebrtc/setup.py", line 45, in run raise e File "/home/ipywebrtc/setup.py", line 38, in run self.distribution.run_command('jsdeps') File "/opt/anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/home/ipywebrtc/setup.py", line 118, in run raise ValueError(msg) ValueError: Missing file: /home/ipywebrtc/ipywebrtc/static/extension.js

Any idea?

AudioRecorder - selectively display recorder (or hide the audio widget) or changing to a HBox layout

I have created an AudioRecorder instance in a jupyter notebook for recording from the microphone.

The jupyter notebook output displays a recorder widget and an audio widget. Is there a way to turn off the audio widget display? I could call recorder.audio to show the audio player widget only , but could not find a way to only show the recorder widget.

I tried:
recorder.audio.layout.display = 'none' and then invoked the recorder object. But it still displays the audio widget.

If I set recorder.layout.display to 'none' then it hides all the widgets. Is there any attribute that will only display the 'record' widget and not the audio? If not, how can this be accomplished?

How to do live processing on a webcam stream

I searched through the examples but it isn't clear how to do live processing of webcam images.
Thanks for your help!

remote jupyter notebook server and recording using camera stream

I am using ipywebrtc on my local jupyter notebook to record an audio. For this I am using the audio option from the CameraStream. This works just fine.

But if I run the Jupyter notebook server on a remote machine (with --no-browser option) and then just export the display on my local machine, the recording does not work. It is probably trying to access the camera/microphone on that remote server.

Is there a way I can still use this library in such remote setup? (ask it to use my local audio/video device)

Widgets non functional in jupyterlab 2.x

After making a fresh install of jupyterlab 2.1.4, was unable to get ipywebrtc to work in lab, worked fine in notebook.

In notebook interface within jupyterlab, got following error.
Error displaying widget: model not found

On the console, errors of the form
blabla "not registered as a widget module"

These appeared to be traced back to the @jupyter-widgets/jupyterlab-manager being at version 2.0.0, which I think is the real source of the problem. Further in the console log there was an error

"No provider for: jupyter.extensions.jupyterWidgetRegistry."

However, the only way I found to get the version of this extension down to 1.x was to downgrade jupyter.

Save image in a loop

I have a bunch of samples which I visualize with the ITK widgets. Saving out a screenshot one by one in a Jupyter notebook does work quite nicely

viewer = view(Cleaned[0].astype('uint8'))
viewer

image_recorder = ipywebrtc.ImageRecorder(stream=viewer, filename='samplename.png')
image_recorder
image_recorder.recording = True
image_recorder.autosave = True

Now I'd like to save screenshots of the 3D visualizations in a loop, but get no output at all.

Data['OutputName_3D'] = [os.path.join(f, sample + '_3D.png') for f, sample in zip(Data['Folder'], Data['Sample'])]
for c, row in Data.iterrows():
        # Start a viewer
        viewer = view(Cleaned[c].astype('uint8'))
        viewer
        # Save out
        image_recorder = ipywebrtc.ImageRecorder(stream=viewer, filename=row['OutputName_3D'])
        image_recorder
        image_recorder.recording = True
        image_recorder.autosave = True

Since I have about 25 samples I'd really like to loop through the saving part and not do it manually...
Can you give me a pointer on how to do achieve this?

Convert audio recording in 'webm' to 'flac' format

I am trying to convert audio recording in webm format to 'flac'
Below is my code:

import time
camera = CameraStream(constraints={'audio': True,'video': False})
recorder = AudioRecorder(stream=camera, format='webm')
recorder.recording = True
time.sleep(5)
recorder.recording = False
recorder.audio.autoplay = False
ipd.display(recorder.audio)
with BytesIO(recorder.audio.value) as f:
audioFile = AudioSegment.from_file(f, format='webm')
with BytesIO() as f:
audioFile.export(f, format='flac')
audioBytes = f.getvalue()

I am getting the following error:

Can anyone suggest where I am going wrong?

"See this tweet" link in README.md is broken

Remove the trailing > in https://twitter.com/maartenbreddels/status/1027995865024262144>

Can't create CameraStream without Camera

Hi, I want to record audio only but I can only do this on computers with a camera device. Is there any work around? Thanks

camera = CameraStream(constraints={'audio': True,'video': False})
camera

This is the output

Error creating view for mediastream: Requested device not found

Missing ipywidgets Video after `pip install`

I tried doing a clean binder setup and pip installing ipywebrtc and get the following error message and see that there is no Video widget in ipywidgets. The setup is running ipywidgets 7.2.1 and the repo/notebook are here: https://github.com/kmader/ivissearch/blob/master/notebooks/SearchDemo.ipynb

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-17-ba437b967460> in <module>()
      2 from skimage.io import imread
      3 mn_model = kapps.MobileNetV2(input_shape = IMG_SIZE, include_top=False)
----> 4 from ipywebrtc import ImageRecorder, CameraStream

/srv/conda/lib/python3.6/site-packages/ipywebrtc/__init__.py in <module>()
      3 import ipywidgets as widgets
      4 from ._version import version_info, __version__
----> 5 from .webrtc import *
      6 
      7 

/srv/conda/lib/python3.6/site-packages/ipywebrtc/webrtc.py in <module>()
     10     Bool, Bytes, Dict, Instance, Int, List, TraitError, Unicode, validate
     11 )
---> 12 from ipywidgets import DOMWidget, Image, Video, Audio, register, widget_serialization
     13 from ipython_genutils.py3compat import string_types
     14 import ipywebrtc._version

ImportError: cannot import name 'Video'

How do I get access to individual camera frames?

I want to use this to do work with OpenCV programming in python. How do I get access to the frames coming in from my camera?

CameraStream odd behaviour on Mac

Hi guys
great work on this, being able to view a camera feed in a jupyter notebook is going to be great for processing images in the lab.
I am getting some strange behaviour on the CameraStream.ipynb example. Audio is set to False but it's definitely on, and actually resulting in an earsplitting reverb. Also the constraints 'video': { 'width': 80, 'height': 60 } do not change the image size.
Cheers
Rob

Is possible to set CamStream size larger than 640*360?

It seems that ipywebrtc limits the size of video to be no larger than 640*360 (maybe ipywidgets does it?)

My camera is actually outputting 1080p, but as long as I set the size over 640*360, it always gives a blank frame of video...

previously i thought it still outputs correct stream of data and it just because of the browser not able to display properly,

but it turns out not after checking out saving pictures, they are blank as well.

post-link script failed for 0.4.0

Upon installing ipywebrtc-0.4.0 from conda-forge I get

LinkError: post-link script failed for package conda-forge::ipywebrtc-0.4.0-py_0

Everything seems to ok for 0.3.0
I use python 3.6 from default channel.

Thanks for your work on ipy*

Azure image_recorder infinite loop

Thank you for the tool.

I want to run it in a azure notebook. The Camera is loaded fine and also displayed, but the image_recorder runs into an endless loop as soon as i execute it, without displaying the buttons. What can I do - or do you know any alternative to load and save images from the webcam into a azure notebook?

Capturing CameraStream raises error "No data, did you record anything?"

I am trying to capture 100 snapshots programatically from my webcam but somehow, the ImageRecorder does not record any image. However, when capturing the images using the dedicated widget button, it works like charm.

This is my code:

First code block (which simply shows the user the webcam input)

from ipywebrtc import CameraStream, ImageRecorder
from time import sleep

camera = CameraStream.facing_user(audio=False)
camera

Second code block (which should capture 100 shots from the camera):

recorder = ImageRecorder(stream=camera)
recorder.recording = True

for i in range(1,100):
    sleep(0.1)
    recorder.save("webcam/" + str(i) + ".png")

Executing the last block, the following error appears:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-77bec100e9f4> in <module>
      4 for i in range(1,100):
      5     sleep(0.1)
----> 6     recorder.save("webcam/" + str(i) + ".png")

c:\program files (x86)\microsoft visual studio\shared\python37_64\lib\site-packages\ipywebrtc\webrtc.py in save(self, filename)
    397             filename += '.' + self.format
    398         if len(self.image.value) == 0:
--> 399             raise ValueError('No data, did you record anything?')
    400         with open(filename, 'wb') as f:
    401             f.write(self.image.value)

ValueError: No data, did you record anything?

What am I missing here?

How to record audio from microphone with AudioRecorder

I want to be able to record the audio coming from the microphone, when using

recorder = AudioRecorder(stream=video)

How can I set my stream to be the microphone?