Write a "how it works" documentation

[vincentfretin]

Write a "how it works" documentation

[2xAA]

Information about the connection/handshake would be nice, so people can easily write adapters :)

[vincentfretin]

The connection to a signaling server and connecting to other participants depends on the WebRTC server tech you use, but I know what you mean. To create an adapter you need to implement a set of methods, you have the list on the interface NoOpAdapter
You can look at the different adapters in this repo to have a sense of what is needed in those methods.

[2xAA]

True, I've just created a slimline WebSocket server as I was finding easyrtc too opaque as an easy "get to grips" solution.

What helped me to understand everything in the end was putting a console.log in each of the existing wseasyrtc adapter's methods to see the exact flow and expected callbacks.

For example, I didn't really understand when a client would be sent an individual message vs a broadcast, and if they should be handled separately (the answer is no, they should both be fed to the messageListener callback in setDataChannelListeners).

Another issue I had was the setRoom function, it seems like the connection and setRoom are independent, but with wseasyrtc at least, the adapter expects the room to be joined on connection, before it returns the clientid - that's not obvious unless you debug quite closely.

I hope this is useful feedback!

[vincentfretin]

This is useful feedback. Putting console.log is something I did a lot too.
You can look at the more simple socketio and webrtc in this repo that doesn't have the open-easyrtc dependency. I kept those in the repo for educational purposes. Those weren't never battle tested on production and may have issues like #243.

You're correct about the clientId of wseasyrtc adapter, you know it only after you're connected to the room, and that's an issue actually, make it hard to fix this issue #320. The janus adapter in comparison you set the clientId before connecting.

individual message:
NAF.connection.sendDataGuaranteed(toClientId, "chatbox", data);
broadcast message:
NAF.connection.broadcastDataGuaranteed("chatbox", data);
see #410

Depending on the adapter the implementation can be different, sending over reliable datachannel of each participant (mesh typology, the easyrtc adapter) or via websocket (wseasyrtc adapter). For janus adapter, broadcast can be on reliable datachannel or websocket, sending a private message you need to use the websocket transport (NAF.connection.adapter.reliableTransport="websocket") and not "datachannel" that doesn't support it, but this one is specific to the way the rust sfu plugin is written.

[tomfelder94]

I also find it quite hard to get a good understanding of the whole flow, when is which method called from the adapters. Also when is information broadcasted to the channel, i.e. in a client-server kind of way, and what information is sent p2p. I will also try to understand better adding a console.log to all the different methods in the adapter. A visual overview/ diagram as you proposed would be much appreciated, whenever you find time.

[vincentfretin]

Alright I'll focus first on the transfer logic for the entities data, I won't go in details on the signaling part in this post.
All the transferred data are the entities, mostly all is handled in the networked component, it uses the special u (update), um (update multiple) an r (remove) messages.

networked-aframe/src/NetworkConnection.js

Line 2 in 3299622

var ReservedDataType = { Update: 'u', UpdateMulti: 'um', Remove: 'r' };

When you connect, each networked component will take care of sending the entity to all participants if you're the owner of the entity. When you create an entity, at first you're the creator and owner of it, creator won't change, owner can change during the lifetime of the entity.
Here when I say 'sending the entity' I mean a json containing the full data or a subset of the data of the components of the networked entity, potentially components of its children, based on the networked schema associated with the networked entity.

networked-aframe/src/components/networked.js

Line 277 in 3299622

this.syncAll(undefined, true);

is called via the onConnected callback registered for the the connected event that is triggered here

networked-aframe/src/NetworkConnection.js

Lines 70 to 76 in 3299622

	connectSuccess(clientId) {
	NAF.log.write('Networked-Aframe Client ID:', clientId);
	NAF.clientId = clientId;
	var evt = new CustomEvent('connected', {'detail': { clientId: clientId }});
	document.body.dispatchEvent(evt);
	}

This call syncAll(undefined, true) meaning send to all participants the owned entities and with data marked with isFirstSync: true, will call

networked-aframe/src/components/networked.js

Line 342 in 3299622

NAF.connection.broadcastDataGuaranteed('u', syncData);

When a user join, all the participants are notified, if those other participants own some entities, they will send their owned entities to the joined user with

networked-aframe/src/components/networked.js

Line 340 in 3299622

NAF.connection.sendDataGuaranteed(targetClientId, 'u', syncData);

This is actually when the user opens a connection to the new user that we do the firstSync to this user only

networked-aframe/src/NetworkConnection.js

Lines 130 to 137 in 3299622

	dataChannelOpen(clientId) {
	NAF.log.write('Opened data channel from ' + clientId);
	this.activeDataChannels[clientId] = true;
	this.entities.completeSync(clientId, true);
	var evt = new CustomEvent('clientConnected', {detail: {clientId: clientId}});
	document.body.dispatchEvent(evt);
	}

that calls

networked-aframe/src/NetworkEntities.js

Lines 141 to 147 in 3299622

	completeSync(targetClientId, isFirstSync) {
	for (var id in this.entities) {
	if (this.entities[id]) {
	this.entities[id].components.networked.syncAll(targetClientId, isFirstSync);
	}
	}
	}

Later, if you take ownership of an entity (with the takeOwnership api), it calls syncAll() to broadcast to all participants the change, it will use

networked-aframe/src/components/networked.js

Line 342 in 3299622

NAF.connection.broadcastDataGuaranteed('u', syncData);

but this time with isFirstSync: falseon the data.

When an entity is removed by the owner, it broadcasts a remove message to all participants:

networked-aframe/src/components/networked.js

Line 581 in 3299622

NAF.connection.broadcastDataGuaranteed('r', syncData);

All the above use the reliable transport. Depending of the adapter it can be the websocket or a WebRTC datachannel configured with { ordered: true }

When you do some changes to a owned entity (like position and rotation or other components defined in the networked schema), it uses

networked-aframe/src/components/networked.js

Line 104 in 3299622

NAF.connection.broadcastData('um', data);

to broadcast to all participants using the unreliable transport, see no Guaranteed in the name of this api.
Depending of the adapter it can be websocket or WebRTC datachannel configured with { ordered: false, maxRetransmits: 0 }.
The data are sent 15 times per sec (NAF.options.updateRate=15) so every 66ms if something changed.

networked-aframe/src/components/networked.js

Line 112 in 3299622

this.nextSyncTime = this.el.clock.elapsedTime + 1 / NAF.options.updateRate;

When you receive a notification that a user disconnected, all non persistent entities he created even if owned by someone else are removed. If there are persistent entities that the user created, each participant will try to get ownership of those persistent entities.
This is this part in the code:

networked-aframe/src/NetworkConnection.js

Lines 139 to 146 in 3299622

	dataChannelClosed(clientId) {
	NAF.log.write('Closed data channel from ' + clientId);
	this.activeDataChannels[clientId] = false;
	this.entities.removeEntitiesOfClient(clientId);
	var evt = new CustomEvent('clientDisconnected', {detail: {clientId: clientId}});
	document.body.dispatchEvent(evt);
	}

that calls

networked-aframe/src/NetworkEntities.js

Lines 155 to 172 in 3299622

	removeEntitiesOfClient(clientId) {
	const removedEntities = [];
	for (var id in this.entities) {
	const entity = this.entities[id]
	const creator = NAF.utils.getCreator(entity);
	const owner = NAF.utils.getNetworkOwner(entity);
	if (creator === clientId || (!creator && owner === clientId)) {
	const component = this.entities[id].getAttribute("networked")
	if (component && component.persistent) {
	// everyone will attempt to take ownership, someone will win, it does not particularly matter who
	NAF.utils.takeOwnership(entity);
	} else {
	removedEntities.push(this.removeEntity(id));
	}
	}
	}
	return removedEntities;
	}

The last participant that took ownership wins.

networked-aframe/src/components/networked.js

Lines 467 to 472 in 3299622

	networkUpdate: function(entityData) {
	// Avoid updating components if the entity data received did not come from the current owner.
	if (entityData.lastOwnerTime < this.lastOwnerTime ||
	(this.lastOwnerTime === entityData.lastOwnerTime && this.data.owner > entityData.owner)) {
	return;
	}

Feel free to contribute a diagram from the above explanation. :) You can start one with https://excalidraw.com and share it here if you want.

[tomfelder94]

Thanks a lot Vincent, that does clarify already a lot of things. Do I understand correctly that with the WsEasyRtcAdapter and EasyRtcAdapter all the data is sent peer-to-peer, even if it is broadcasted to everyone? I read the following doc, is it correct that there is no adapter in place which which sends the data from a client to a server, which then broadcasts the messages through websockets to each client? We are looking for a solution that we can scale with many users, by using a commercial solution like Ably, so we do not need to maintain our own server. It is not quite clear to us how we would implement such an adapter.

[vincentfretin]

Indeed with the broadcastData and broadcastDataGuaranteed API to broadcast data to everyone, depending of the adapter, can be sending to each participant peer to peer (mesh topology) or once to a server that broadcasts to all participants (SFU topology).

Only EasyRTCAdapter is peer-to-peer (audio/video/data with WebRTC), mesh topology.
WsEasyRtcAdapter is data only via WebSocket, so client->server->clients.
Janus adapter is a SFU, uploading only once audio/video with WebRTC to a server, data via WebRTC or WebSocket (you can choose). With janus adapter you can have 20-25 users with audio in a room without issue.

[tomfelder94]

That makes sense, thanks. I will implement an Ably adapter and see if I find time to create a diagram along the way. That was all very helpful!

[tomfelder94]

I created an AblyAdapter today which seems to work but I still have a few questions:

• I am connecting to ably in the adapters connect() method. The broadcastDataGuaranteed(dataType, data) method is called immediately after, ideally we could await a promise that is resolved once connection is established. I looked at how it is done in the socketio adapter, there you are simply checking if the socket already exists or not. Is that the way to do it?

networked-aframe/src/adapters/naf-socketio-adapter.js

Lines 170 to 183 in 3299622

	broadcastDataGuaranteed(type, data) {
	const packet = {
	from: this.myId,
	type,
	data,
	broadcasting: true
	};
	if (this.socket) {
	this.socket.emit("broadcast", packet);
	} else {
	NAF.log.warn('SocketIO socket not created yet');
	}
	}

• What are the shouldStartConnectionTo(clientId) and getConnectStatus(clientId) methods exactly here for? It seems like they are not needed for the AblyAdapter.
• Do I understand correctly, that the three "stream" methods (see below) are only used if we stream media using WebRTC?

    startStreamConnection(clientId) 
    closeStreamConnection(clientId) 
    getMediaStream(clientId) 

[vincentfretin]

For easyrtc, the check is done in the open-easyrtc library
https://github.com/open-easyrtc/open-easyrtc/blob/221f6439a083aa6e47b0e508d155723b60bdec4e/api/easyrtc_int.js#L3251-L3253

There is a similar check in janus adapter for publisher WebRTC connection
https://github.com/networked-aframe/naf-janus-adapter/blob/fed7925cd1f4f771980f63a845edcfd606ae3544/src/index.js#L1016-L1019

In theory you should never see those warnings, If you see them, there is probably something wrong.
Now that you mention it, I have sometimes the warning with the janus adapter but never looked why, although the code in the adapter seems to do the right thing.

As we saw above, the initial broadcastDataGuaranteed call is triggered from the this.syncAll(undefined, true) call from the connectSuccess callback set via

networked-aframe/src/NetworkConnection.js

Lines 46 to 49 in 3299622

	this.adapter.setServerConnectListeners(
	this.connectSuccess.bind(this),
	this.connectFailure.bind(this)
	);

In the case of the janus adapter, the socket is opened, the publisher WebRTC is created and then the connectSuccess callback is called. So you definitely should do something like that if your reliable transport is via WebRTC datachannel
https://github.com/networked-aframe/naf-janus-adapter/blob/fed7925cd1f4f771980f63a845edcfd606ae3544/src/index.js#L246-L252
The broadcast for the first sync via the WebRTC datachannel initially is probably useless for janus sfu, because nobody is connected to us yet, so the real first sync will happen when we're connecting to each participant and when the onOccupantConnected callback is called
https://github.com/networked-aframe/naf-janus-adapter/blob/fed7925cd1f4f771980f63a845edcfd606ae3544/src/index.js#L344

For the easyrtc adapter, the connectSucess callback is called by the library after the websocket is opened
https://github.com/open-easyrtc/open-easyrtc/blob/221f6439a083aa6e47b0e508d155723b60bdec4e/api/easyrtc_int.js#L6085
the broadcast is done peer to peer:

networked-aframe/src/adapters/EasyRtcAdapter.js

Lines 166 to 175 in 7d7b1ab

	for (var roomOccupant in roomOccupants) {
	if (
	roomOccupants[roomOccupant] &&
	roomOccupant !== this.easyrtc.myEasyrtcid
	) {
	// send via webrtc otherwise fallback to websockets
	this.easyrtc.sendData(roomOccupant, dataType, data);
	}
	}
	}

It will actually send the first sync via websocket if the WebRTC datachannel is not yet ready.
https://github.com/open-easyrtc/open-easyrtc/blob/221f6439a083aa6e47b0e508d155723b60bdec4e/api/easyrtc_int.js#L3273-L3280

[vincentfretin]

For the shouldStartConnectionTo(clientId), startStreamConnection(clientId), closeStreamConnection(clientId) api
some adapter may use those methods to implement the connection and disconnection logic to participants, this is based on the occupantsReceived callback:

networked-aframe/src/NetworkConnection.js

Lines 82 to 108 in 3299622

	occupantsReceived(occupantList) {
	var prevConnectedClients = Object.assign({}, this.connectedClients);
	this.connectedClients = occupantList;
	this.checkForDisconnectingClients(prevConnectedClients, occupantList);
	this.checkForConnectingClients(occupantList);
	}
	checkForDisconnectingClients(oldOccupantList, newOccupantList) {
	for (var id in oldOccupantList) {
	var clientFound = newOccupantList[id];
	if (!clientFound) {
	NAF.log.write('Closing stream to', id);
	this.adapter.closeStreamConnection(id);
	}
	}
	}
	// Some adapters will handle this internally
	checkForConnectingClients(occupantList) {
	for (var id in occupantList) {
	var startConnection = this.isNewClient(id) && this.adapter.shouldStartConnectionTo(occupantList[id]);
	if (startConnection) {
	NAF.log.write('Opening datachannel to', id);
	this.adapter.startStreamConnection(id);
	}
	}
	}

The easyrtc adapter uses those methods:

networked-aframe/src/adapters/EasyRtcAdapter.js

Lines 127 to 150 in 7d7b1ab

	shouldStartConnectionTo(client) {
	return this._myRoomJoinTime <= client.roomJoinTime;
	}
	startStreamConnection(clientId) {
	this.easyrtc.call(
	clientId,
	function(caller, media) {
	if (media === "datachannel") {
	NAF.log.write("Successfully started datachannel to ", caller);
	}
	},
	function(errorCode, errorText) {
	NAF.log.error(errorCode, errorText);
	},
	function(wasAccepted) {
	// console.log("was accepted=" + wasAccepted);
	}
	);
	}
	closeStreamConnection(clientId) {
	this.easyrtc.hangup(clientId);
	}

janus adapter doesn't use those methods and do the logic internally:
https://github.com/networked-aframe/naf-janus-adapter/blob/fed7925cd1f4f771980f63a845edcfd606ae3544/src/index.js#L819-L825

Those methods are used to subscribe to a participant for naf updates, audio, video.
In the case of easyrtc a WebRTC connection to a given participant is created to receive the naf updates via datachannel, audio and video tracks.
For janus sfu, it's also creating a WebRTC connection for each participant, although going really through the server, not peer to peer.
For another sfu like LiveKit or mediasoup where it's using only two WebRTC connections (one to publish, one to receive), it can be used to subscribe to a participant datachannel, audio track and video track.

getMediaStream(clientId, streamName) api is used for audio and video, it's used in the the networked-audio-source and networked-video-source components.