automerge/trellis

Trello clone / sample app for Automerge persistence library

Trellis

This is a largely defunct research project left online for posterity. If you're discovering this now, you should take a look at PushPin, a much more complete & robust implementation of this approach to software development.

Trellis is a Trello clone built as an Electron desktop application. We built Trellis to demo MPL, a distributed persistence library that allows developers to build collaborative applications that are realtime, offline, and network partition tolerant.

Motivation

Our motivation to build Trellis was to create an application that would demonstrate the unique features of MPL. We chose to build a Trello clone as our demo application due to Trello's popularity as a web-based project management tool, its highly collaborative design, and its "dogfooding" usefulness so that we could use Trellis as we're building it.

Like most modern web applications, the original Trello relies on a stable Internet connection in order to function. With MPL, we can extend Trello's functionality to work offline, collaboratively with peers over a local network, and with synchronization so that peers coming back online can merge their changes with other peers using automatically merging data structures.

Features

Trellis includes many of the basic features of Trello:

• Create cards
• Create lists
• Drag cards from one list to another
• Comment on cards
• Assign cards to individuals

In addition, Trellis includes:

• Offline Mode: Trellis is a desktop application and works offline with no network connection. Your changes will be saved locally and sync automatically when you re-establish an Internet connection.
• Realtime Collaboration: When online, Trellis will sync changes between peers in realtime.
• Automatically Merging Synchronization: Trellis uses MPL for data persistence, which uses automatically merging data structures for synchronization.
• Network Partitions: If you are disconnected from the Internet but share a local network with a peer, you can continue to collaborate in realtime with peers on the same network.

Conclusions

We designed Trellis as an experiment to test and demonstrate the functionality of MPL, a distributed persistence library built on top of Conflict-free Replicated Data Types (CRDTs). Our goal was to assess CRDTs to see whether these data structures, which can be automatically merged, also work well in practice. We expected there may be several issues with using CRDTs in practice, such as CRDTs becoming unperformant at even small scales, or that the developer experience might be too difficult for the typical app developer unfamiliar with CRDTs.

To our surprise, MPL was reasonably performant at our scale and we were able to design a developer interface that is intuitive, familiar to developers using modern frontend libraries, and does not require developers to understand or interface with CRDTs. The end-user experience was also impressively free of data corruption: as long as data updates are done with as small of a "granularity" as possible there are very few instance of data merging in unexpected ways and users always converged to the same application state. The remaining challenges at the conclusion of this project were mostly in building a robust and decentralized networking layer for CRDT synchronization.

Setup

You can download the latest version of Trellis from the releases page.

To test peer synchronization:

• Mac OSX: Make a copy of your Trellis.app and start both clients.
• Windows: Start Trellis.exe twice.

You should see both peers in the Network panel of both clients. If you now open the same document in both clients, you'll be able to see the edits from one synchronize to the other in realtime.

Development

Prerequisites

You will need the Electron Forge CLI installed globally:

$ npm install --global electron-forge

Windows only: You also need to have the C++ compiler toolchain installed. The easiest way to do this is to install Microsoft's windows-build-tools from an elevated prompt (run as Administrator).

$ npm install --global --production windows-build-tools

Alternatively, you can install Visual Studio Build Tools, following the steps described here.

Linux only: You also need to have the required GTK development headers and libraries installed:

$ sudo apt install libgtk2.0-dev  # example for Ubuntu 19.04

Running and testing the app

To start the app:

$ npm install
$ npm start

To run tests:

$ npm test

Distribution

To generate a packaged application:

$ npm run package

• In Mac OSX, this generates a trellis.app package located in the out/trellis-darwin-x64 directory. To run it:

  $ open out/trellis-darwin-x64/trellis.app
  

• In Windows, this generates a Trellis.exe application, with associated resources, in the out/trellis-win32-x64 directory. To run it:

  $ ./out/trellis-win32-x64/Trellis.exe
  

• In Linux, this generates a Trellis ELF executable (dynamically linked). To run it:

  $ ./out/Trellis-linux-x64/Trellis
  

trellis's Issues

Title bar sometimes changes back to 'index.html'

Bigger drop targets

The UX of dragging cards isn't that fun because of small drop targets. Ideally it should work like this:

Changes are lost if you quit before autosave window

Steps to reproduce:

1. Assign someone to a card by clicking their avatar
2. Press cmd-Q (on Mac) right after
3. Re-run the app, open the file, and the assignment is lost

Any reason not to save on every change, rather than on a timer?

Peergroup / Cannot read property 'id' of undefined

Was working on a document with two machines connected then did Document -> New on one and saw this message. Can add steps to reproduce if desired.

Merging assignments unexpected results

If I merge two boards with assignment updates on the same card, the changes seem to get lost.

Steps to reproduce:

1. File -> Open -> `ampl-week-immutable.trellis
2. File -> Save As -> adam.trellis
3. click Martin's avatar on "Team Summit" to add him to assignments
4. wait at five seconds for autosave
5. File -> Open -> ampl-week-immutable.trellis
6. File -> Save As -> pvh.trellis
7. click Roshan's avatar on "Team Summit" to add him to assignments
8. File -> Merge -> adam.trellis

What I expected: both Martin and Roshan are assigned to Team Summit

What I got: only Roshan is assigned to Team Summit (merged change from adam.trellis seemed to be dropped)

adam.trellis and pvh.trellis

unexpected peer on first launch

Launched it for the 1st time
a) there's a document added by Meriweather
b) meriweather is listed as a peer

I assume that this is just some startup data but this also implies that the Meriweather peer can see changes I make, which would be not good.

refusing to continue with app not knowing who can see my work or why meriweather is showing up as a peer.

fast-toggling network causes desync

Toggling the network on and off quickly leaves things in a broken state. I'm not sure why, but somehow during peer.send() the connection goes from "open" to "closed" between when we call connected() and when we try to send()

Steps to reproduce:

• open two clients (same machine is fine)
• connect them and see that changes are propagating
• choose one and click the network toggle quickly off and back on

Flickering activity lights

A nice feature would be to show network activity by having the green LEDs next to Introducer, Bonjour, and peers flicker (i.e. dark green to bright green) when sending or receive messages.

Fork with peers doesn't leave the group

Steps to reproduce:

1. Open Trellis on machine 1, make a new document, share it to clipboard
2. Open Trellis on machine 2, open that same document
3. On machine 1, Document -> Fork
4. Make changes on machine 2

Got: changes show up on the forked document on machine 1

Expected: forking disconnects me from peers and not receive changes from them anymore, unless they follow me to the new document.

Syncing gets stuck and needs a kick

When two or more clients are connected, it often happens that client A has changes that client B hasn't shown yet. This can usually be fixed with a "kick" in the form of client B making one change, which causes all of client A's changes to flood in.

Haven't yet gotten good steps to reproduce although running the demo video script very often gets stuck on this, especially after reconnecting from offline edits.

Also worth noting that this has popped up periodically in our group testing scenarios for the last week or two, so it's probably not a new problem.

Can't see peer names

I also can't click to set my name, but apparently I have a name set because when I make a change on the board it shows properly in the changes list.

License type

Just wondering how this is intended to be licensed.

better network logging

Opening the web console currently shows a continuous stream of messages from the network:

This makes it really difficult to debug potential networking errors. For example, right now it seems like there's a bug where one client continuously applies to same delta and gets stuck in a loop but it's hard to assess because there's a stream of messages even when there's no user activity. Could we filter it down to only when a client sends/receives deltas?

Clicking "connect" while connected causes trouble

1. Open two Trelli.
2. On the non-introducer, click "connect".

ID collision causes merged cards to overwrite each other

Steps to reproduce:

1. File -> New
2. Add card -> "Card A" -> Add
3. File -> Save -> "a.trellis"
4. File -> New
5. Add card -> "Card B" -> Add
6. File -> Merge -> a.trellis

Card A doesn't show up, because Card B overwrote it. They share the ID of 5.

missing peerStats entry leads to trouble

Again related to reconnect lifecycle stuff; we seem to delete your peerStats but not reinitialize it correctly on reconnect. I haven't tracked down why this is yet.

Nunito (card font) doesn't work on some systems

You can see the fallback font here:

https://camo.githubusercontent.com/276f561978372700c3812175db319490929d453e/68747470733a2f2f692e696d6775722e636f6d2f705347746e336d2e706e67

This happens on my Mac laptop but my iMac has the correct font.

DocID in titlebar and DocInspector don't match

When forking or creating new documents, the DocId seems to fall out of sync. In this example, cobalt-moscow-70 vs violet-shanghai-73.

Steps to reproduce:

1. Document -> New

Note DocID mismatch.

MaxListenersExceededWarning

I just connected two machines to a fresh document and saw this. Will work on steps to reproduce if desired.

amplnet / Failed to execute 'send' on 'RTCDataChannel'

Got this error (and several more like it) when I had two machines connected to a document, then the other machine switched away to a different document and back to this one. The UI is now completely frozen on the machine with the error.

Switching documents causes trouble

1. Open two instances.
2. Move one to another document.
3. Bring other one to that document.
4. Make a change.

Make time travel mode more clear

Time travel is currently a mode that puts the entire board into read-only mode, but there's almost no indicator that this is the case.

A half-finished design idea for this is here:

In addition to making time travel mode obvious, exiting it should be easier. Probably clicking anywhere in the right sidebar or pressing Esc should exit.

package it on FDroid

Hi,
would you please package this app and publish it on FDroid, so that everybody can use it on their android phone? Your colleagues (scientists) and client audience (software developers) will get to experience how great automerge works at one tap.

https://f-droid.org/en/docs/Submitting_to_F-Droid_Quick_Start_Guide/

Issue #50 clarifies it's MIT licensed and therefore suitable for F-Droid and other free software distributions.

Offline mode: fonts come from the internet

When I ran Trellis on the plane:

Doesn't stop the app from working but seems out of character with our offline-first approach.

TypeError: nextProps.network.peerStats.get is not a function

Happens on any run of Trellis.

Syncing gets stuck when clicking a new document in Documents list

Steps to reproduce:

1. Run this script

#!/bin/sh

mkdir ~/alice
rm ~/alice/*.trellis
mkdir ~/bob
rm ~/bob/*.trellis

PORT=4200 INTRODUCER= SAVE_DIR=~/alice NAME="Alice" npm start &

sleep 5
echo '-------'
echo
echo "Document -> New in the Alice client. Press enter when done..."
read

PORT=4201 INTRODUCER=localhost:4200 SAVE_DIR=~/bob NAME="Bob" npm start &

2. After the Alice client launches, execute Document -> New
3. Press enter in the console
4. From the Bob client, click on the document that Alice is in (in the screenshot: silver-shanghai-8)
5. Bob client gets stuck with a blank board and no changes that Alice make are reflected

Possibly important clue: if you wait until Bob is already connected to Alice at the time Alice creates the document, syncing works fine. The document has to be created before Bob connects to Alice.

Runaway messages

When disconnecting from the network then reconnecting, both clients sometimes go into a runaway message loop:

New document doesn't stream in from peers on network

Steps to reproduce, two machines:

1. launch Trellis A, Document -> New
2. launch Trellis B, connect to peergroup via introducer or Bonjour
3. Trellis A -> Share to clipboard
4. get the URL, Trellis B -> Open

Trellis B is left with a blank document:

To simulate this locally, just remove the Trellis file ("local cache") before launching the second Trellis, like this:

1. launch Trellis A, Document -> New
2. delete the file ~/Documents/Trellis/$docid.trellis
3. connect to peergroup via localhost:4242 or somesuch
4. launch Trellis B

Blank document, as above.

Changes pane not displaying anything

Steps to reproduce:

1. Document -> New
2. Make some changes

Nothing appears in the Changes pane:

Can't read property toJS() of undefined on time travel

From trellis://cobalt-tokyo-30 clicking any change produces:

Cannot read property 'messagesSent' of undefined

Got this by running two local clients, setting network toggle off on one, making changes to both, then reconnected to network.

leaving a peer group

@orionz @pvh I'm storing the network docId's in the .trellis files, so that whenever you open a .trellis file it'll join the corresponding peer group. I need a way to leave or reset the current peer group and join a new one, but I couldn't figure out how to do that since everything uses the same singleton webrtc object currently.

Cannot read property 'title' of undefined

Happens pretty regularly with today's build:

Usually reproducible by having a second machine connect to the document.

Confusing UX when opening a document with no saved data and no peers to fetch from

I'm often confused when opening a URL with no data attached. Steps to reproduce:

1. Open a Trellis URL (say, trellis://gray-istanbul-10) without any a local copy (~/Documents/Trellis/gray-instanbul-10.trellis) or any peers who can send me the document. Best way to do this is just make something up, i.e. trellis://not-a-document

What I got: the app appears to open it, but it's a blank screen that just looks broken.

What I expected: some feedback that I have no data for this document. Effectively it is an invalid URL until I can connect to someone with a copy.

Press spacebar while hovering over a card to assign yourself to it

Mouse hover + spacebar = assign yourself like in Trello.

This would require that we know who "you" are. Right now there's the peerName but that has no connection to the hardcoded avatars on card assignments.

Indicate conflicting changesets in Changes pane

Would be really useful if conflicting changesets (that is, something like two users concurrently changing a card title while disconnected from each other) would show an indicator. Maybe a different color change node?

The CRDT library has a conflicts API but it may need some changes in order to associate conflicting changes with an exact changeset and no others.

Escape from file dialog give error

Steps to reproduce:

1. File -> Open
2. press escape

I expect this to cancel the dialog and go back to the app.

Change highlight on app open is confusing

When you open the app the last change highlights. This is confusing, presumably that change happened a long time ago.

Steps to reproduce:

1. Document -> New
2. Add a card titled "Highlight me"
3. Quit Trellis
4. Run Trellis again

What I got: the "Highlight me" card highlights when I re-open the app

What I expected: It should highlight after I make the change, but not when re-opening the app. Same for changes that come in over the network.

Can't use Trellis once network is turned off

Steps to reproduce:

1. Connect to a document, no problem making changes
2. Click network toggle in upper right to disconnect
3. Try assigning a card or adding a new card

Got result: can't make any changes and this error appears in the console

Expected result: I can work offline and reconnect later