Giter Site home page Giter Site logo

capital-g / webrtcgui Goto Github PK

View Code? Open in Web Editor NEW
12.0 1.0 0.0 3.16 MB

SuperCollider GUI for browsers with remote sync capablities.

License: GNU General Public License v2.0

Dockerfile 1.33% JavaScript 2.39% HTML 3.51% Vue 27.28% SuperCollider 13.36% TypeScript 52.13%
gui webrtc supercollider

webrtcgui's Introduction

webRTCgui

webRTCgui allows to publish, modify and sync dynamic GUI elements / parameters via the internet (or local network) from within SuperCollider.

Screen recording

This GUI parameters syncs across multiple devices, also across the internet.

As SuperCollider can not directly communicate via WebRTC we need a translation layer to do the network lifting. To get a better understanding we quickly explain each component involved.

Service Folder Description
Server js/server.ts A express server which serves the frontend and distributes the necessary messages for communication. You can run this your local machine or on a remote machine (server) if you want to publish the GUI via the internet.
Frontend js/src A vue.js 3 frontend / website through which clients can interact and view the available parameters. Should run on the same machine as the backend and in default the server cares for this.
Client js/client.ts Our translation layer between SuperCollider OSC messages and the WebRTC server. This needs to run on your local machine.
SuperCollider ./classes A simple Quark with the class WebRTCGUI. Should be used on your local machine.

Setup

SuperCollider

Before we start with setting up the services you should install the necessary Quark in SuperCollider by entering the following lines into the editor and evaluate them.

Quarks.install("https://github.com/capital-G/webRTCgui.git");
// recompile interpreter to make new classes available
thisProcess.platform.recompile;

WebRTC

Clone the repository to a folder of your choice via

git clone https://github.com/capital-G/webRTCgui.git
cd webRTCgui

The easiest way to set up all necessary services is using Docker with docker-compose.

Local server and client

This assumes that sclang is started on port 57120 which can be verified by executing NetAddr.langPort or can be changed by setting e.g. export SC_PORT=57121.

In order to try it out on your local machine you can type

docker-compose up --build

Once everything spun up you can access the frontend on http://localhost:3000 and start creating controls from SuperCollider by reading the docs

HelpBrowser.openHelpFor("Classes/WebRTCGUI")

Other users within your network can also access the website via http://<your-local-ip>:3000.

To understand the connections better here is a schematic on how the services are connected in a local setup.

Local setup

Remote server and local client

Assuming you want to have the website accessible via the internet you need a remote machine on which you can to deploy the server on which you should run

# set a password for the server and start the server
BACKEND_AUTH_TOKEN=someBetterPassword \
docker-compose -f docker-compose.server.yml up --build

By appending a -d flag the service will start in the background.

Make sure you have setup the network properly so the service is reachable by either having port 3000 forwarded in your firewall (e.g. ufw) or setting up a reverse proxy (see below). If you can see the website you have set it up properly.

After this you need to start the client on your local machine which acts as a translation layer between our local SuperCollider instance and the remote server.

# set the password and address from the server and start the local client
BACKEND_AUTH_TOKEN=someBetterPassword \
BACKEND_ADDRESS="http://my.ip:3000" \
docker-compose -f docker-compose.client.yml up --build

If everything worked it should say

webrtcgui-client-1  | Connected to server

and from there on you can continue with the SuperCollider documentation (see above).

Here are the schematics for a remote workflow with reverse proxy.

Remote setup

Setup a nginx reverse proxy

Using a reverse proxy allows you to secure the application better and also add supports for https certificates to the server which will improve usability across devices.

Here is an example of such a nginx config file which would redirect the traffic from my-domain.com to our server.

server {
  server_name   my-domain.com;

  location / {
    proxy_pass  http://localhost:3000;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "Upgrade";

      add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload";
      client_max_body_size 0;

      access_log /var/log/nginx/remote.access.log;
      error_log /var/log/nginx/remote.error.log;
  }
  listen 80;
}

Modify it according to your URL and port mappings, restart nginx and let certbot give you a certificate for your config.

After this you can verify that you can access the server with your browser which should display the frontend website.

Authentication

Please do not use precious passwords as these passwords are transmitted non-encrypted and may get spilled by storing and printing them in plaintext.

To restrict the creation and deletion of GUI elements there needs to some kind of authentication to distinguish between the local client which is attached to SuperCollider and the remote clients, accessing the server via a browser. You can set the environment variable BACKEND_AUTH_TOKEN in the server and client shell to the desired password, e.g.

export BACKEND_AUTH_TOKEN=somePassword

Development

For development we do not rely on the docker environment because it is rather nasty to sync the node_modules folder between container and host machine. Currently this project uses Node 14 with typescript which makes the development of the communication much more safe as we use type checks on the messages we send and receive.

To install all necessary dependencies execute

cd js
npm install

and start the setup via

npm run local-dev

The Vue development server is then available under http://localhost:8080, the server starts under port 3000.

Please install and setup pre-commit before committing which will run some linting checks.

The schematics of the dev environment.

Dev schematic

License

GPL-2

webrtcgui's People

Contributors

capital-g avatar

Stargazers

Paul Osetinsky avatar Martin  avatar Yuhei avatar Andrew Grathwohl avatar Vinzenz Aubry avatar igor medeiros avatar  avatar Jack Armitage avatar Mads Kjeldgaard avatar  avatar Christophe Lengelé avatar Andrew Ikenberry avatar

Watchers

 avatar

webrtcgui's Issues

Add schematcis via uml

First sketch via https://www.planttext.com/

grafik

@startuml

rectangle "Local" {
  rectangle "docker" as localdocker {
      frame client {
        rectangle "client.js:57220" as LocalClient
      }
    }
    frame "SuperCollider" {
      rectangle "sclang:57120" as sclang
    }
}

rectangle "Server" {
  rectangle "docker" {
  frame "backend" {
    rectangle ":3000" <<express>> as express
  }
  frame "frontend" {
    rectangle ":8080" <<vue>> as vue
  }
  }
  
  frame "nginx" {
    rectangle ":80" <<webserver>> as webserver
  }

}

cloud "Remote Clients" as RemoteClients


RemoteClients <-> webserver : "WebRTC"


vue -> webserver : "http"
express <-> webserver : "ws"


LocalClient <--> webserver : "WebRTC"

sclang <-> LocalClient : "OSC"


@enduml

Implement locking mechanism

A rabbit hole, but at least it should

  • prevent feedbacks
  • prevent out-of-sync GUIs (maybe send a "beacon"/state of the server?)

It seems the more delay/lag we have via network the worser it gets

Implement RCE workflow

by providing a nice interface for remote code execution this could become a full alternative gui for SC?

Combine back and client into one folder

Currently we have 3 different nodejs environments.
Vue is really heavy and we can ignore it via a build anyway, but the other two could be combined to make everything a bit more straightforward

Combine expressjs and static vue build

Makes it harder to debug so maybe only for deployment but would make the routing a bit easier and should also increase performance as the dev server is really slow iirc.

Fix slider entering

@mouseon/@mouseoff seem to work good in theory but they do not work on touch interfaces and also don't work fast (enough?) when one just clicks on a section instead of dragging it.

Share documents

Allow to share documents, combined with #18
But this does not resolve how to handle execution?

Add JITlib constructor

Implement something along WebRTCGui.jitGui which deploys a (tabbed?) view for

  • Ndefs
  • Tdefs
  • Pdefs (see #21 )

This needs to pick up updates from Ndefs etc.

A sketch for tabbed ndefs

(
var ndefs = ();
Ndef.all[\localhost].activeProxies.do({|p|
	ndefs[p] = Ndef(p);
});

y = WebTabLayout((
	\ndef: WebTabLayout(ndefs),
));

w.layout = y;
)

Add auth for client

Currently everyone can send/modify the sliders on the page - by adding a simple password to the communication between server/local client we can restrict this to our local client.

Get values of controller on startup

Currently only what controllers are available is synced at startup but not their position.
This could probably done for each controller item independently in order to make syncing easier.

Implement log scaled sliders

Currently the sliders in the frontend offer only a linear scale - would be nice to offer e.g. log scaled as well to reflect ControlSpec of SC

Add Pdef component

Maybe something like a form would be cool so its easier to manipulate the Pdef

Add button

Allow to use Buttons to (de)-activate certain behavior

Make socket.io host address dynamic in frontend

Currently the remote deployment does not work because we build against the wrong socket address, see

const socket: Socket<ServerToClientEvents, ClientToServerEvents> = io(process.env.VUE_APP_SOCKET_ENDPOINT || "http://localhost:3000");

2 ways to solve this

  • inject the URL as argument during build time
  • use the same host url as the current visited url - maybe this is a good fallback for the first usecase

GUI does not sync across devices

We currently ignore the values that we receive from other clients in the frontend. This bug was introduced by the rewriting in typescript which changed the behavior of how we distribute the receiving channels.
Maybe its best to fix this properly by adding a state management?

Docker compose client fails under linux

- SC_HOST=${SC_HOST:-host.docker.internal}
- SC_PORT=${SC_PORT:-57120}
- BACKEND_ADDRESS=${BACKEND_ADDRESS:-http://host.docker.internal:3000}

host.docker.internal is a macOS/Windows featuer, for linux see https://stackoverflow.com/questions/48546124/what-is-linux-equivalent-of-host-docker-internal

Creating webrtcgui_client_1 ... done
Attaching to webrtcgui_client_1                                                                                                    
client_1  |  
client_1  | > [email protected] client /root/web_rtc_gui 
client_1  | > ts-node --esm src/client.ts 
client_1  |  
client_1  | Start client 
client_1  | Backend address: https://remote.dennis-scheiba.com 
client_1  | SuperCollider address: host.docker.internal:57120 
client_1  | Client port: 57220 
client_1  | Auth token: "somePW" 
client_1  | Started client 
client_1  | Connected to server 
client_1  | Received freq: 113.30357142857143 
client_1  | Received freq: 119.75 
client_1  | Received freq: 132.98214285714286 
client_1  | Received freq: 152.32142857142856 
client_1  | Received freq: 167.92857142857142 
client_1  | Error: getaddrinfo ENOTFOUND host.docker.internal 
client_1  |     at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:71:26) { 
client_1  |   errno: -3008, 
client_1  |   code: 'ENOTFOUND', 
client_1  |   syscall: 'getaddrinfo', 
client_1  |   hostname: 'host.docker.internal' 
client_1  | } 
client_1  | npm ERR! code ELIFECYCLE 
client_1  | npm ERR! errno 1 
client_1  | npm ERR! [email protected] client: `ts-node --esm src/client.ts` 
client_1  | npm ERR! Exit status 1 
client_1  | npm ERR!  
client_1  | npm ERR! Failed at the [email protected] client script. 
client_1  | npm ERR! This is probably not a problem with npm. There is likely additional logging output above. 
client_1  |  
client_1  | npm ERR! A complete log of this run can be found in: 
client_1  | npm ERR!     /root/.npm/_logs/2022-12-15T16_03_32_530Z-debug.log 
webrtcgui_client_1 exited with code 1

Make colors configurable

Set primary, secondary color etc via SC. Needs to find a nice way how to pass these colors to e.g. a Button - maybe WebRTCGui.primaryColor.

Add forwarding of wss in reverse proxy

Currently it seems the wss connections are rejected by the server, see

Firefox can’t establish a connection to the server at wss://remote.dennis-scheiba.com/socket.io/?EIO=4&transport=websocket&sid=_2cYrudD3-U8XHojAAAG.

Thankfully socket.io falls back on a simple get/post workflow so it still works.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.