typeintandem/tandem

Typing in Tandem. Decentralized, cross-editor, collaborative text-editing!

Tandem

Tandem is a decentralized, collaborative text-editing solution. Tandem works with native text editors, works across different editors, and uses peer-to-peer connections to facilitate communication.

Tandem exists as a set of plugins for native text editors. We currently support Sublime Text 3 and Neovim. We also unofficially support Vim.

Collaborating is as easy as installing the plugin on your editor and creating a Tandem Session. Invite other people to your session, and get typing in tandem!

Installation

Requirements

Tandem is currently supported on Mac OS X - it may work on Linux, but use at your own risk.
To use Tandem, your environment must have python3.6+ and node.js installed (tested and confirmed working on node.js v7.7.4).
python3.6+ is a requirement for our agent code and node.js is required by our CRDT.

Plugins

Please follow the installation guides for your plugin of choice:

• Sublime
• Neovim
• Vim (unofficially supported)

How it Works

Tandem is split into four components: the editor plugins, the networking agent, the conflict-free replicated data type (CRDT) solution, and the rendezvous server.

Editor Plugins

The plugins interface with the text buffer in your plugin, detecting local changes and applying remote changes, and allowing users to create, join and leave sessions. Each plugin has its own repository and code.

Agent

The agent establishes connections between other peers connected to your collaborative session. It takes messages from the editor plugin and broadcasts them to all peers, and with the help of the CRDT instructs the editor to apply remote changes to your local text buffer.

CRDT

The CRDT is used to represent the state of your local document, and transforms document edits into operations that can be applied remotely, without conflicts. We submit these conflict-free operations to other peers via the agent.

Instead of reinventing the wheel and writing the CRDT ourselves, we leveraged the work done by GitHub's team working on Teletype. We used their CRDT under the hood and instead focussed our efforts on integrating the CRDT with different editors.

Rendezvous Server

The rendezvous server is used to help establish peer-to-peer connections. It server notes the connection details of any peer that joins a session. When any subsequent peer wants to join, the server provides the connection details of all other peers in the session so that they can communicate directly with each other.

Note: Peer-to-peer connections cannot always be established. As a convenience to the user, the rendezvous server will also act as a relay to send data between agents when this happens.
If you wish to disable this behaviour, you can do so by disabling the USE_RELAY flag in: agent/tandem/agent/configuration.py

Terms of Service

By using Tandem, you agree that any modified versions of Tandem will not use the rendezvous server hosted by the owners. You must host and use your own copy of the rendezvous server. We want to provide a good user experience for Tandem, and it would be difficult to do that with modified clients as well.

You can launch the rendezvous server by running python3 ./rendezvous/main.py. Change the address of the rendezvous server used by the agent in the configuration file to point to your server's host. This file is located at: agent/tandem/agent/configuration.py

License

Copyright (c) 2018 Team Lightly

See LICENSE.txt

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Authors

Team Lightly
Geoffrey Yu, Jamiboy Mohammad and Sameer Chitley

We are a team of senior Software Engineering students at the University of Waterloo.
Tandem was created as our Engineering Capstone Design Project.

tandem's Issues

Not sure how this is supposed to work.

I installed the deps and the neovim plugin. Restarted neovim.

In one shell I opened neovim foo.txt and did :Tandem (copied session-id )and wrote some stuff into the buffer

In another shell, I opened neovim . and did :Tandem sessionid.

I got shown an empty buffer but nothing was reflected in the other buffer, no matter what I wrote in the 1st neovim session's buffer.

I'm on High Sierra (osx).

Neovim (0.3.3) healthcheck is all fine (python2 & 3 support)

I don't see any errors but it doesn't look like it's working. Am I missing something?

Here are the logs:

tandem-agent-62043.log
tandem-agent-62367.log
tandem-crdt-debug-1546944835838.log
tandem-crdt-debug-1546944888109.log
tandem-crdt-1546944888109.log
tandem-crdt-1546944835838.log

vim.command("enew") is broken

After you run a Tandem session and stop it with "TandemStop", restarting it causes this issue:

Error detected while processing :
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 54, in start
  File "/Users/sameer/Documents/development/tandem/plugins/vim/tandem_lib.py", line 285, in start
    self._initialize()
  File "/Users/sameer/Documents/development/tandem/plugins/vim/tandem_lib.py", line 70, in _initialize
    vim.command('enew')
vim.error: Vim(enew):E37: No write since last change (add ! to override)
Press ENTER or type command to continue

Looks like the command is broken. Probably need to add "!" to overwrite it.
As a side note, I had deleted all content on my buffer, so it probably didn't even need to do this in the first place.

add support for a TeX/LaTeX editor with Tandem

I'm extensively using real-time collaborative edition for TeX/LaTeX documents.

Today, this usage is dominated by Overleaf/Sharelatex which are both somehow proprietary.

Yesterday, there was bluelatex (full disclosure: I was one of the author, together with @tdurieux and @satabin), but the project has stalled, in particular because of serious bugs / limitations in the synchronization.

It would be great to have collaborative edition for TeX/LaTeX documents coming again in the free world, and having Tandem support in a Linux TeX/LaTeX editor (eg TeXWorks) would be a solution!

Add python3 version requirement

When running python3 ./agent/main.py using Python 3.4 I run into errors with the trailing commas after kwargs that pop up throughout the repo. With python 3.6 it runs fine.

Could you add the minimum version for python and nodejs to the readme?

List port and host used in README

To allow easier firewall configuration, it would make sense to have a section the README detailing the traffic generated by tandem.

Like:

• TCP connection to rendezvous.foo.com on port 443 using HTTPS
• UDP connection to the either the remote host on a dynamic port or the rendezvous server

Also, what happen if the two hosts are on different local subnets that are directly connected, is a direct local connection tried first?

Visual Studio Code support

VS Code is one of the popular editors around, and is being adopted by a lot of developers around the world. Would be good to see support for the same.

Vim plugin optimization

The input_checker in the VIM plugin currently polls every 1s. We can change this to use autocommands and turn this into event driven logic.

question: plan for a web client?

Is there a plan to have Javascript web client? Is that on the roadmap?

[Sublime Text] Send initial buffer contents on host start in Sublime

[Sublime Text] Python3 not found

Launching a session will silently fail if you try to do so before sublime loads certain environment variables.

We try to run our agent python3 and invoke the version of python that is bundled with Sublime, but it can't be found. This is not a major issue right now, but should probably be fixed before release. It occurs every time the application (Sublime) is relaunched, and is not a one-time only thing.

Communication between two agents

Future extension possibilities

• Show cursor locations of remote participants
• End-to-end encryption of inter-agent messages
• Show selections of remote participants
• Support multiple buffers in one session
• Define the editor messages using protobufs
• Improve serialization of messages (probably just move everything to protobufs)
• Use a build tool
• Support more editors
  • Atom
  • VSCode (?)
  • IDEs (?)

Add plugin bindings to allow for "direct connections"

The agent still supports the ConnectTo message, but the vim/neovim plugins no longer send it because we use the hole punching messages.

It may be useful to have a plugin binding that sends this message so we can establish a "direct connection" (e.g. for debugging or if your editing group members are all routable under a private network). This will also require the plugin to display the port number that it's listening on.

License is not DFSG-complaint

Hi,

Unfortunately, due to the following clause:

     (e) Any Derivative Works may not be configured to use the default server
         provided by the Copyright holders. You must configure all Derivative
         Works to use servers owned by You, or servers You are otherwise
         authorized to use.

... typetandem is not in compliance with (at least) the Debian Free Software Guidelines. It would also not be compliant with the FSF's Four Freedoms.

This is a shame as typetandem is genuinely cool but Debian (and others) simply cannot distribute it. :(

Use GUI controls to start/stop Tandem in Sublime

Communication between agent and plugin

[Sublime] Create and merge fork of Package Control

The link to our sublime repo should be added to the Package Control channel, and we should open a PR to merge it in.

Not launch blocking but will help increase # users and ease of installation.

Investigate concurrency issues

• Add logging to node process to get visibility into errors
• Shorten length of patches that are pure insertions and pure deletions (to allow concurrent edits near the same region)
• Investigate why sometimes the index to point mapping exceeds the bounds of the buffer
• Build some kind of fuzzer to facilitate testing concurrent edits
• Use sync calls in neovim instead of async_call

Connection between agent and plugin

Implemented by #34

Support for jetbrains suite?

Launch preparations

• Investigative and preparatory work
  • Investigate options for deployment (e.g. package both node/python3, one or the other, or neither) and pick an approach
  • Relevant build scripts to package dependencies
  • Manual smoke tests on common platforms (see which are currently supported)
    • macOS (we expect it to work)
    • Linux (we think it should work)
    • Windows (we don't expect it to work)
    • Cross platform testing (i.e. see if participants on different platforms can edit a document together)
    • Make a decision on which platforms to support based on test outcome and any bugs that arise from testing
• Plugin release
  • Create public repository for vim plugin
  • Create public repository for neovim plugin
  • Create public repository for sublime text plugin
  • Write plugin release scripts that will commit to the public repositories
• Tandem website
  • Register a domain
  • Set up analytics
  • Build an initial landing page
  • Set up some kind of issue tracking system for public feedback (e.g. report a bug -> sends an email -> creates internal GitHub issue)
  • Write the needed documentation to tell people about how to install and use Tandem
  • Update website with download links, documentation, and feedback/bug reporting
  • Create a video to showcase Tandem (? - Teletype has one)
• Tandem rendezvous server (dependent on outcome of #61)
  • Set up AWS deployment of the rendezvous server
  • Configure DNS appropriately if needed
  • Add server side monitoring/analytics so we have visibility into its usage
• Tandem plugin rendezvous
  • Display session uuid in a user friendly way
  • Change "connect to" to accept a session uuid instead of host/port

NAT traversal using hole punching

• Build experimental hole punching proof-of-concept server + client just to see if we can exchange packets via UDP
• Switch the agent's transport over to UDP
• Build the tandem server to facilitate setting up sessions and for rendezvous
• Modify the agent to communicate with the tandem server
• Stretch goal: Allow set up of a session with multiple participants
• Add interagent keep alive messages
• Add a rendezvous <-> agent heartbeat message

Allow multiple users to edit document (plugin side)

Currently the plugin designates one user as the host and others as the listener (user must designate) in order to prevent listeners changes from being echoed back to the host.

We should allow all users to be hosts/listeners in the sense - this will be related to and aided by #40 where we don't send the entire buffer contents but rather the diffs b/w buffers.

Create Atom Plugin

Should have connection and communication between agent and plugin

Emacs support?

website domain expired?

hello!

what's up with the project? :) it looks like the main domain i had in my links has expired:

http://typeintandem.com/

...and there hasn't been a lot of activity in the last year or so... is this still going?

thanks!

Create Sublime Plugin

• Create Base Plugin
• Bring up to speed with nvim changes

Bonus:

• Move shared utilities b/w nvim and sublime to a common lib folder.

Only one buffer should be affected within a session

Even if multiple buffers are open, Tandem should only be listening to changes and writing to one of them.

Needed for both Sublime/Neovim

[Sublime Text] Disable menu options when appropriate

See if we can disable the Tandem options when appropriate:

• should not start/join session when one is running
• should not leave session until one has been started/joined

Also see if we can do this with command palette commands

Connection between two agents

Create Vim Plugin

Have plugin send/apply diff operations

Currently we broadcast the entire buffer contents / apply the whole buffer.
We should send/accept diffs (what has changed between cursor movements) to the agent/CRDT process.

Build error

After cd crdt, yarn, yarn build I get:

Version: webpack 3.12.0
Time: 1865ms
    Asset    Size  Chunks                    Chunk Names
bundle.js  911 kB       0  [emitted]  [big]  main
   [6] ./stores/document.js 161 bytes {0} [built]
 [100] multi ./index.js 28 bytes {0} [built]
 [101] ./index.js 1.05 kB {0} [built]
 [112] ./api/index.js 458 bytes {0} [built]
 [113] ./api/apply_operations.js 365 bytes {0} [built]
 [114] ./api/get_document_text.js 257 bytes {0} [built]
 [115] ./api/get_document_operations.js 270 bytes {0} [built]
 [116] ./api/replicate_document.js 279 bytes {0} [built]
 [117] ./api/set_document.js 235 bytes {0} [built]
 [118] ./api/set_text_in_range.js 344 bytes {0} [built]
 [119] ./io/index.js 590 bytes {0} [built]
 [120] ./utils/logger.js 492 bytes {0} [built]
 [136] ./node_modules/logform ^\.\/.*\.js$ 1.12 kB {0} [built]
 [138] ./node_modules/logform/dist ^\.\/.*\.js$ 508 bytes {0} [built]
    + 189 hidden modules

ERROR in ./node_modules/winston/lib/winston/transports/http.js
Module parse failed: Unexpected token (186:6)
You may need an appropriate loader to handle this file type.
|     }
|     const req = (this.ssl ? https : http).request({
|       ...this.options,
|       method: 'POST',
|       host: this.host,
 @ ./node_modules/winston/lib/winston/transports/index.js 42:11-28
 @ ./node_modules/winston/lib/winston.js
 @ ./utils/logger.js
 @ ./index.js
 @ multi ./index.js
error Command failed with exit code 2.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Remove rendezvous server from plugin READMEs

The rendezvous server is not distributed as part of this plugin/agent. It should be removed from the license section of each plugin's README, and a link to this monorepo's README should be provided instead.

Move shared logic to lib/

Move logic common to sublime and neovim editors to a lib directory

[Sublime Text] Open new View when connecting to host

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet.
We recommend using:

• CircleCI
• Travis CI
• Buildkite
• CodeShip
• Azure Pipelines
• TeamCity
• Buddy
• AppVeyor
  But Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.