invisible-college/tawk.space

Social video chats

tawk.space

Social video chats

Website: https://tawk.space/

The concept

Tawk is built around the idea of a salon. No, not those places you get haircuts, but 18th century French Enlightenment salons. Aristocratic men and women gathered to discuss politics, art, literature, and more during this "age of conversation".

Now, in the 21st century, the internet can bring us to a new "digital age of conversation". While text chats are often plagued with anonymous trolling, forcing users to face their peers produces deeper, more satisfying conversations.

Tawk is meant to be simple and flexible to use.

• There is no login--simply share a link to your space (https://tawk.space/your-space).
• Each space can have multiple chat groups. Drag your video to create one or mouseover another group to hear what they're saying. This is an excellent way to create "breakout" groups.

Obligatory thanks

Tawk was made possible through several helpful projects:

• Statebus, a realtime database
• Agora.io, the video backend
• DesignEvo free logo designer

Embed in your website

Insert this code into your html somewhere:

<script src="https://download.agora.io/sdk/web/AgoraRTC_N-4.1.0.js"></script>
<script src='https://tawk.space/hark.js'></script>
<script src="https://invisible-college.github.io/diffsync/diffsync.js"></script>
<script src="https://tawk.space/client/shared.coffee"></script>
<script src="https://tawk.space/client/tawk.coffee"></script>

And if you aren't using statebus already, include this too:

<script src="https://stateb.us/client6.js"></script>

Now you can place a TAWK widget anywhere on a statebus page like this:

TAWK({name: 'username', space: '/', height: 500, width: 500})

Set up your own tawk instance

Clone tawk.space and install dependencies:

[email protected]:invisible-college/tawk.space.git
cd tawk.space
npm install

You must set up certs for https in a folder called certs/ in the tawk.space repo. This will be used by both tawk.space and statebus. Put the public key in certs/certificate and private key in certs/private-key.

sudo ln -s /etc/letsencrypt/live/<your-domain>/fullchain.pem certs/certificate
sudo ln -s /etc/letsencrypt/live/<your-domain>/privkey.pem certs/private-key

Finally, run it! Again, you might want to use screen.

sudo node index

Your tawk instance is running at https://your-domain

Contributing

PR's and issues are very welcome!

tawk.space's Issues

Group loses background color after somebody moves out of it

^The first group is white while the others are (correctly) gray

Tawk needs unit tests

For now I had the client buses in mind. There isn't a whole lot else to unit test because tawk is mainly render functions.

Tawk should indicate whether people are muted

...And should also use a microphone icon rather than a volume icon.

Need a collaborative design space in tawk (e.g. iframe of Figma)

Muffle the other groups, not just lower their volume

We can muffle other groups with a low-pass filter, which is a built-in feature of the HTML5 audio API. :)

This will be awesome! We can make other groups sound like they are underwater, like wub wub, and be able hear when they are laughing but not get distracted with what they are saying.

Crash in presence.coffee

I got a crash in tawk.space -- looks like in presence.coffee -- which seems to be freezing the UI so that when a new person joins, you can't see them.

Here's the crash:

Uncaught TypeError: Cannot read property 'getBoundingClientRect' of undefined
  at guess_cursor_location (https://tawk.space/client/presence.coffee:188:26)
  at t.dom.CURSOR (https://tawk.space/client/presence.coffee:130:9)
  at t.render (https://stateb.us/client6.js:13:7083)
  at t.<anonymous> (https://stateb.us/client6.js:13:1196)
  at t [as render] (https://stateb.us/client6.js:1:10671)
  at t.<anonymous> (https://stateb.us/client6.js:3:24328)
  at t.i [as _renderValidatedComponent] (https://stateb.us/client6.js:5:15756)
  at t.<anonymous> (https://stateb.us/client6.js:3:23431)
  at t.i [as updateComponent] (https://stateb.us/client6.js:5:15756)
  at t._performComponentUpdate (https://stateb.us/client6.js:3:23012)
  at t.performUpdateIfNecessary (https://stateb.us/client6.js:3:22627)
  at t.receiveComponent (https://stateb.us/client6.js:3:10486)
  at t.receiveComponent (https://stateb.us/client6.js:3:23235)
  at a._updateChildren (https://stateb.us/client6.js:5:12404)
  at a.updateChildren (https://stateb.us/client6.js:5:12119)
  at a._updateDOMChildren (https://stateb.us/client6.js:4:2795)
  at a.<anonymous> (https://stateb.us/client6.js:4:1396)
  at a.i [as updateComponent] (https://stateb.us/client6.js:5:15756)
  at a.performUpdateIfNecessary (https://stateb.us/client6.js:3:10727)
  at a.receiveComponent (https://stateb.us/client6.js:3:10486)
  at a.receiveComponent (https://stateb.us/client6.js:4:1191)
  at t.<anonymous> (https://stateb.us/client6.js:3:23471)
  at t.i [as updateComponent] (https://stateb.us/client6.js:5:15756)
  at t._performComponentUpdate (https://stateb.us/client6.js:3:23012)
  at t.performUpdateIfNecessary (https://stateb.us/client6.js:3:22627)
  at s (https://stateb.us/client6.js:5:26015)
  at r.perform (https://stateb.us/client6.js:6:11602)
  at o.perform (https://stateb.us/client6.js:6:11602)
  at o.perform (https://stateb.us/client6.js:5:27662)
  at Object.<anonymous> (https://stateb.us/client6.js:5:27878)
  at Object.i [as flushBatchedUpdates] (https://stateb.us/client6.js:5:15756)
  at r.closeAll (https://stateb.us/client6.js:6:12300)
  at r.perform (https://stateb.us/client6.js:6:11685)
  at Object.batchedUpdates (https://stateb.us/client6.js:4:15136)
  at Object.c [as enqueueUpdate] (https://stateb.us/client6.js:5:26670)
  at t.forceUpdate (https://stateb.us/client6.js:3:24093)
  at https://stateb.us/client6.js:13:2146

Here's the outcome when a new person joined (you can't see them) and then I opened a new window which made a new version of me (michael) who also didn't appear in the old window.

@tkriplean -- maybe you have an idea bout this bug.

Scaling isn't right for groups > size 4

The scaling works perfectly for 1, 2, 3, or 4 people. But when I have 5 people in a chat, there's extra whitespace over on the right. And when I have 8 people, it gets reallllyyy tiny:

That's a bit too small.

Tawk needs screensharing

Janus gives us an api for this: https://janus.conf.meetecho.com/screensharingtest.html as long as we're fine sending either camera video or screenshare video but not both. Sending both would require multiple streams. Also note publishers switching from video to screensharing would need to unpublish their own stream and publish a new one. That might "just work" on the subscriber side.

You would need to download an extension. We can reuse Janus's extension: https://chrome.google.com/webstore/detail/janus-webrtc-screensharin/hapfgfdkleiggjjpfpenajgdnfckjpaj

Tawk should combine video streams on the server

This is more of a long term issue, but it'll be good to keep it in the back of our minds.

Essentially we need a real time version of this:
https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos

Right now every client downloads every stream separately, which is really inefficient in terms of bandwidth. We should package several (or all) streams together.

Occasionally localstream never loads

The symptom is that you're local stream always has the spinner gif, and never loads. If you're the first person in the space, you start in your own group, then magically get kicked out back to the topbar (me.group gets set to null somehow).

I thought 9d48d33 would fix this, but I still occasionally (like 1/20 times) still see it.

Scratch space should highlight links

The most common use of the scratch space (so far) has been to paste links. We should make them <a href="..."> links so that people can click on them.

It's hard to tell when media links are slow

There are a couple of user stories here:

1. While my local stream appears to be near perfect, it's choppy/frozen to others. I keep talking confidently, without feedback that others can't really see/hear me. I want to be able to tell when others perceive me as choppy/frozen.
2. I see choppy video streams for others, and I don't know why it happens--maybe tawk.space is slow. I want feedback like the media link's bitrate to tell me what's going on.

Is starting outside the space confusing?

It seems like when you open tawkspace, your attention is drawn to the people already there, rather than your own tiny box in a corner. It takes a while to figure out that you have to move yourself.

We could instead get rid of the topbar entirely (except for the textbox with your name?), and mandate that users have to be in a group. You start out in your own group, and you can join other groups.

Thoughts?

Group scratch space should be a collaborative editor

Right now if two people are editing the group scratch space, they end up clobbering each other. This is confusing behavior.

Ideally, each group scratch space can have multiple people at once. You should be able to see others' cursors, selections, etc. Essentially a mini-google doc.

@morgandixon can I steal your implementation of this?

When you mouse over a group, they should also hear YOU

I just cleaned up the audio volume calculations: 552ceae

It's working well for us in considerit. When I mouse over a buddy's group, I can hear them. However, they cannot hear ME! This is gonna be confusing, because in real life, whenever you can hear someone, they can probably hear you too. Mousing over a group would be a great way to say "Hey guys, wanna come join my group?"

So I think we need to pass through the mousover group information to the server, because other people need to know when you are mousing over them so that they know to increase your volume. Maybe we should design this at the same time as making groups globally editable (#10)? Do these designs interact?

Other WebRTC libraries?

I've been talking with @canadaduane who is building a game that runs in a web page (with WASM + Javascript), and is trying to add a WebRTC video chat to it. I've been telling him what I know about the available tech, but I think @karth295 knows a little more than me.

We've had some issues with video chat bugs, and although some of them seem to be browser bugs (because they crash the browser, require a browser restart to fix, or only appear on some browsers), I also saw some bugs in Meetecho's product that looked like Meetecho bugs, because the UI itself just wasn't working.

So we have been wondering about other WebRTC libraries. Karthik mentioned Twilio, which I think would be cool to try. Duane also just made this list. He is interested in trying out pion. It looks like it's 2 years old.

Open Source:
Pion - seems to be most popular
Janus - built by meetecho & chosen by tawk.space
Simple Peer - built for p2p and most popular in that category
Kurento - another popular server
WebRTC-Experiment - lots of popular experiments with WebRTC, reliability unknown

Closed Source:
Whereby - Chris' suggestion
Twilio - Michael's suggestion

Noteworthy:
Complex State Machine
Popularity of Various WebRTC Projects on Github

Participants sometimes can't hear each other

We don't know why, but sometimes some participants can hear one person, while others can't. That person's microphone does work, and tawk sets the person's volume to the correct number (20, 80, or 100%). Here are debugging tips:

1. Everybody should check that their microphones are on and working: https://www.onlinemictest.com/. Obviously if some people can hear but not others this isn't the problem.
2. When you hover over someone it shows you the percent volume that their audio is playing at. If that number if wrong, tawkspace has a bug. The group volume code is limited to the dom.GROUP and dom.PERSON functions if you want to look there, but the code looks right to me...
3. Maybe there are weird semantics around people joining/leaving groups, or changing the volume on video streams. It would be helpful to figure when that happens.
4. Very unlikely, but we could be finding browser bugs. I'm not sure how to debug that.

I'm inclined to think it's #3. If that's the case, we could leave everybody's volume at 100% (across all groups) and see if that fixes it.

Tawk doesn't work in Edge

You can see yourself, but not others.

Theory: we use a deprecated syntax for <video> tags

video.src = URL.createObjectURL(media_stream)

When we should be using:

video.srcObject = media_stream

Can't hear if somebody joins while you're on a different tab

Browsers do not autoplay videos while you are in a different tab. This is mainly so that websites don't play ads in other tabs, you can open multiple youtube videos, etc.

For tawk, this means that if somebody joins your group, their <video> tag can't automatically play. You can't tell that they joined, and they can't get your attention.

Tawk should send a user desktop notifications (similar to Slack) to inform them that a user joined. These notifications should only fire if you are not actively looking at tawk.

Recording video/uploading into a dedicated repository

Is there any documentation on recording and storing tawk.space videos?

Text field below a group should automatically resize to fit the text

I'd like to describe what I'm currently doing in my group (my office), and update this over time.

But I have to manually resize the text field, such as when I'm updating what I'm doing.

Likewise, I'd like to read what other people are up to, but if they've written something longer, I have to manually resize.

This problem is compounded because whenever someone joins or leave a group, the manual text height setting is reset.

Tawk Space should adjust lighting

I often use tawk in a relatively dark room, and people can't see my face very well. I usually compensate by turning on a lamp, facing the light, or increasing the brightness on my computer.

Tawk should just preprocess the video and change the brightness for me (given that the original stream is not so dark that it doesn't have enough information regardless of brightness).

Try out Janus's simulcast or SVC

From their videoroom demo: https://github.com/meetecho/janus-gateway/blob/master/html/videoroomtest.js#L405

Here's the SVC demo: https://janus.conf.meetecho.com/vp9svctest.html.

For background:

• https://webrtcglossary.com/simulcast/
• https://webrtcglossary.com/svc/

SVC seems waaay more scalable and practical than #15

Sometimes Travis's video stream gets delayed

This isn't a big deal, because his audio is still in realtime and in sync, but if I watch his video, I'll see his mouth move like a couple seconds after the words were emitted. :P

Hear yourself when speaking (with headphones)

Old wired telephones will loop the speaker's mic signal back through the earpiece, so the speaker can hear themselves speak. This gives the speaker a sense of how their voice is being picked up by the microphone, and sent to the other side. Humans naturally use their own ears as feedback for how their voice is being received. (Consider that someone wearing earplugs is likely to speak too loud, because they are listening to their own voice, and can't tell how loud it is.) Musicians also set up a special monitor speaker or headphones to hear themselves, when live on stage. This feedback is critical for knowing how you're received.

Many of the bugs in video chat are due to not knowing how you are being received. We sometimes have our microphones off, and try to talk, without knowing that the other party can't hear us. We often have our mics ON while there's loud background noises, that distract everyone else on the chat without us realizing. We might breathe into the mic, or rub it with our fingers, and creating gross scratching noises that everyone else hears blare and we ourselves are unaware of, because there's no monitor for our audio.

Internet/cell phone audio chat and cell phones have stopped looping the mic audio back through the earpiece to prevent feedback. It's harder to cancel out feedback when you aren't sure how much latency there is on the signal (vs. analog wires), and when you're sending audio over the internet, you don't know how much latency there will be. So they just turn it off.

However, if you're using headphones—which any serious video chatter should—then there's no feedback possible, and it's very useful to have a monitor again. Serious video chatters would benefit from a mic monitor along with nice headphones.

So we should implement a mic monitor loop that's available to people wearing headphones.
I'm not sure what the UI design should be, yet. Perhaps we can start with just an advanced feature (accessed via the console or a tiny menu or keybinding) and then figure out the UI after we experiment with it.

Scratch space (SYNCAREA) should show cursors

It's hard to tell where you can/should type while others are also editing text.

Users should be able to move others around

Currently you can only move yourself in tawk. I can think of a couple reasons to move others:

1. A new user is confused and they're in their own group where nobody can hear them. An experienced user can help them out by moving them to a group.

2. Create a side conversation with somebody (one of the original use cases of tawk)

Users shouldn't be able to modify others' /connection objects directly, so we can create a new piece of state: /groupmod.

sb["/groupmod/user_id"] = new_group_id

The server will update the user's connection object, and broadcast the change to everybody.

Unclear why videos take so long to load

When I first open up tawk, it takes at least couple seconds for me to see my video stream and everybody else in the room. It gets worse when you're on a slow connection and the handshaking takes a long time.

Currently, tawk says that the remote stream is "muted" with no video until getting the remote stream. Instead, it should show something like "loading" until it gets the video stream.

Chrome doesn't round the video's corners anymore

Weird, Chrome now fails to round the videochat corners properly:

That's Chrome on the left and Firefox on the right.

Is Chrome the new IE?

Video should be toggleable and/or default to off

Video streams are expensive: requiring significant bandwidth (~300kbps per stream) and can't tolerate jittery streams very well. We should allow people to turn their video off, and replace it with a picture or avatar.

@ppham suggested that he'd leave tawk open in the background if it were audio-only, but wouldn't want to broadcast his video unless he was actively using it.

Tawk needs UI integration tests

E.g. dragging myself causes an empty group to appear.

This has been a big cause of bugs -- introducing one feature subtly breaks something else.

Lots of feedback lately

Something seems to have changed recently (changes in chrome? change in janus?) and we get lots of feedback when tawk.spacing. I even get some feedback (sounds like a "reverb" effect) when I'm alone in a tawk.space and using headphones, so it seems like it's not even physical feedback but some kind of internal feedback.

Users should be able to reorder themselves within a group

And when joining a group, they should be able to drag to a specific position.

Turning off video also disables audio

The issue is that the client doesn't render a audio or video tag if the person doesn't have video. The logic of dom.PERSON looks something like this:

if not person.hasVideo:
  DIV
    name
    if person.hasAudio
      '(Audio-only)'
    else
      '(Muted)'
else:
  VIDEO

We should just have a video tag in all cases, and if the person doesn't have video, display the message.

Watch youtube videos in sync

This is pretty low hanging fruit -- and is a big reason people use rabb.it.

Tawk needs tests around audio/video

E.g. play fake audio and video and make sure the client can see/hear it.

We might be processing audio repeatedly

We use the Janus events onlocalstream and onremotestream to both display a video stream on the page, and periodically display a green sound bar depending on the audio level in that stream. We use a library called hark to get the decibel level for the stream, and ask it to give us a callback every 200ms.

I set debug: true in Janus.init and added a log statement in receive_stream in tawk.coffee to see whether those callbacks get called multiple times. And indeed they do.

Creating PeerConnection
janus.js:1749 Preparing local SDP and gathering candidates (trickle=true)
janus.js:1781 Handling Remote Track
tawk.coffee? [sm]:585 Calling hark for f25a824b-08f4-4e6d-a8a8-701161a63164 MediaStream
janus.js:1789 Adding onended callback to track: MediaStreamTrack
janus.js:1781 Handling Remote Track
tawk.coffee? [sm]:585 Calling hark for f25a824b-08f4-4e6d-a8a8-701161a63164 MediaStream
janus.js:1789 Adding onended callback to track: MediaStreamTrack
janus.js:1799 Remote track flowing again: Event
tawk.coffee? [sm]:585 Calling hark for f25a824b-08f4-4e6d-a8a8-701161a63164 MediaStream
janus.js:1799 Remote track flowing again: Event
tawk.coffee? [sm]:585 Calling hark for f25a824b-08f4-4e6d-a8a8-701161a63164 MediaStream

You can see the underlying events more clearly below -- the stream gets muted/unmuted temporarily. Maybe due to packet loss?

janus.js:1799 Remote track muted/removed: Event {isTrusted: true, type: "mute", target: MediaStreamTrack, currentTarget: MediaStreamTrack, eventPhase: 2, …}
janus.js:1799 Remote track flowing again: Event {isTrusted: true, type: "unmute", target: MediaStreamTrack, currentTarget: MediaStreamTrack, eventPhase: 2, …}

When this happens, we call hark.hark() multiple times for every stream, and it will give us callbacks from different instances of the same stream. We should instead call hark.stop() on an old stream before starting a new one

Auto-expanding group scratch space doesn't push groups down

Start out with four people in separate groups:

Type in the first box, and this happens:

Either the box should stop expanding, or it should push the next row of groups down

Tawk.space freezes sometimes when scratch text area needs to expand

Behavior:

• tawk.space tab freezes entirely sometimes when scratch text area needs to expand
• I can still hear other people, but they can't hear me
• the freeze never resolves; I need to kill the tab

Replication:

• I've successfully reproduced this bug three times in a row by copy/pasting a large chunk of text into the scratch space that forces resizing during a four-person tawk
  • if I resized the text area first to accommodate the pasted text, the freezing did not happen
  • if I am in tawk space by myself, I'm not able to reproduce the problem with the above strategy
• I've encountered the bug ~5 times previously in a two-person tawk when I was just holding down the enter key in the scratch space

Keyboard focus stolen by scratch area in Safari

Tawk works in Safari now, which is cool!

But I've noticed a little bug -- it repeatedly focuses the scratch area. This happens most noticeably when you try to edit your name in the "My name" textbox, because you can barely type a letter before your keyboard focus moves into the scratch area.

So my name up there is currently "i" because that's all I had managed to type out of "Mike":

:-P

I wonder if this is a bug in the AUTOSIZEBOX widget. I haven't yet looked at the code.

Oh, and in mobile safari, you might notice this as the page zooming into the scratch area, because it thinks that you want to type into it, even though you're just trying to talk to people and want to see the big picture so that you can see their faces.

Debug P2P network disconnects

We just had a 3-way tawk.space with Travis, Myself, and Ethan, where suddenly Ethan could no longer see or hear me:

1. Ethan was the last person to join. He joined in a new solo group, then moved himself to the combined group of Me and Travis.
2. Soon after he joined our group, my picture froze for Ethan, and he could not hear me. However, all other audio/video streams were working.
3. I moved myself out to make my own group. Then I disappeared altogether for Ethan, and Travis and I could still see and hear everyone.
4. Then I reloaded the page, and all streams went back up.

@karth295 Can you help me generate some hypotheses for what went wrong here? Is it true that there was a P2P stream from my client to Ethan's client, and perhaps that stream went down?

I'm happy to own this issue (just self-assigned) if I can ask you questions and get your expertise.

Allow users to pick from multiple cameras.

My thinkpad has two cameras: an IR camera, and a normal camera.

The IR camera is camera 0 so it is used by default in most situations. I have to go into settings to pick a different camera.

tawk.space does not seem to support changing what camera you use.

Visualize network hiccups inline

The chrome notifications were a quick way to show us network issues, but we don't need to demand the user's attention for this information. It would be better to have an ambient visualization of the network strength.

For instance, we could set some state for each notification:

slowness = fetch('slowness')
slowness.all ||= []
slowness.all.push(
  when: new Date()
  what: "local"
)

And render these with icons:

  # Maybe this goes in top bar?
  for slowdown in fetch('slowness').all
    fade = (new Date() - s.when) / fade_time
    IMG
      src: 'hourglass.png'  # or something
      style: opacity: fade

And clean them up periodically with:

fade_time = 5 * 60 * 1000  # 5 seconds
clean_slowness = ->
  slowness = fetch('slowness')
  slowness.all = (s for s in slowness.all when new Date() - s.when < fade_time)
  save(slowness)      
setInterval(clean_slowness, 1000)