RFC: Backstroke Migration

[backstrokeapp]

RFC: Backstroke Migration

[1egoman]

Update: I've secured the domain backstroke.co. My hope is use this domain name instead of backstroke.us for the new deployments as it makes maintaining backward compatibility easier. Any GET requests to backstroke.us would 301 redirect to backstroke.co, and any POST requests would be handled in the old fashion. Here's an updated diagram of the architecture:

Also, I've mostly finished work on the legacy backstroke service (now to be hosted at backstroke.us. The code can be found here.

[1egoman]

I finished up the deployment repository. https://github.com/backstrokeapp/deployment

[1egoman]

The deployment happened at 3pm EDT on October 7th, 2017. The service was down from 3pm to 3:10pm.

I'm glad that this new stuff is finally deployed. Over the next week or so I expect for a few issues to come in with scenarios that I didn't take into account when working on the new stuff, but all in all, I'm pretty satisfied with this release.

Dashboard: https://github.com/backstrokeapp/dashboard/releases/tag/v2.0.0
Server: https://github.com/backstrokeapp/server/releases/tag/v2.0.0
Legacy: https://github.com/backstrokeapp/legacy/releases/tag/v2.0.0
Worker: https://github.com/backstrokeapp/worker/releases/tag/v2.0.0
Deployment: https://github.com/backstrokeapp/dashboard/releases/tag/v2.0.0

In a few weeks, I'll complete the migration by taking down all the old stuff on Heroku, and close this issue.

[m1guelpf]

I have a paid instance of deployhq.com, so if you want I can "donate" the deploy tool and then you can stop worrying about that 😄

[1egoman]

@m1guelpf Are you talking about for the current deployment, or with the new architecture I'm proposing? I'll do some research this weekend and see it would work be helpful for the current state of affairs (I'm unfamiliar with the service) but I think for the new architecture I'd like to try out some sort of immutable deployment such as Docker (and at a cursory glance, DeployHQ doesn't seem immutable). Thanks!

[m1guelpf]

@1egoman I was talking about the old one. I don't think it supports Docker or any other type of immutable deployment... :sad:

[1egoman]

Cool. I'll do some research this weekend. 😄

[1egoman]

@m1guelpf It doesn't look like DeployHQ works with Heroku, so thank for the offer, but I don't think it'll be helpful for maintaining the current Backstroke version 🙁

[eins78]

@1egoman It sounds like https://zeit.co/now does what you want?
Maybe @rauchg wants to "pitch" in? ;)

[1egoman]

@eins78 Interesting, I'll do some research and see if now could fit Backstroke's needs.

[1egoman]

I love the idea of now, but I'm having trouble deploying this repository.

https://zeit.co/rgausnet/server/xvoeisygwy shows that the container takes a very long time to start, though the logs seem to show that the container actually started previously. Once the container did start, I get repeatedly get 502s: https://server-xvoeisygwy.now.sh/

@rauchg Any help you could provide with this?

Also sent a support email to [email protected]:

Hello,

I'm thinking about deploying my open source project Backstroke (https://github.com/1egoman/backstroke) on Zeit. I seem to be having some issues with the deployment.

zeit.co/rgausnet/server/xvoeisygwy shows that the container takes a very long time to start, though the logs seem to show that the container actually started previously. Once the container did start, I get repeatedly get 502s: server-xvoeisygwy.now.sh

For reference, here's the Dockerfile for the container that I'm trying to deploy: https://github.com/backstrokeapp/server/blob/master/Dockerfile, and here's the repository: https://github.com/backstrokeapp/server.

Thanks for the help!
Ryan

[1egoman]

I've spent the last week or so writing a deployment script for Backstroke. My current plan is to host all services on a DigitalOcean droplet, with each service running within docker. In the near term, I plan to use docker-compose to spin up all services on server start since I'm not too concerned with scaling right off the bat. (If I want to scale the service further, I might try nomad.)

My goal was to run all these services on the smallest size droplet (1 core, 512mb ram) but it looks like that is going to be near impossible. Between haproxy, docker and two node processes, the instance runs of of memory within a couple minutes. I'm now running on the 2nd-smallest droplet (2 core, 1gb ram) and I can run docker, redis, and three node processes (worker, server, and legacy) with about ~150mb of ram left over.

I'd prefer to rely on a third party service for hosting the database rather than do it myself, though depending on cost it may make sense for me to just figure it out on my own. Currently, I'm relying on a Heroku free-tier database with a 10,000 row limit (and linking to it externally from the server container) but this is far from optimal.

Unfortunately, this means that I'm going to be spending a bit of of pocket for now - hopefully this new version will gather some more Gratipay donations and can be self sufficient!

While the deployment scripts aren't ready to open source (of course, with secrets redacted), I'll post a link once they are ready. All the existing services in the diagrams above are also now hosted at backstroke.co - api.backstroke.co, app.backstroke.co, and backstroke.co. The legacy service is also hosted at legacy.backstroke.us, though this will eventually be aliased to backstroke.us. Feel free to try them out - I'd love to get feedback on all the work I've been doing over the past 6 months.

I'm hoping these updates provide transparency into how Backstroke's upcoming release is shaping up. Are these helpful? Thanks for using Backstroke! ❤️

[1egoman]

A number of helpful things have happened since the last update:

1. I was able to secure a sponsorship from DigitalOcean! For the next year, at least, hosting shouldn't be an issue. There's now a nice sponsorship note on the new website. Also, this means that I've upgraded to a 2gb droplet, on which the systems perform much better.
2. A complete local development environment has been set up in the https://github.com/backstrokeapp/deployment repository. Now, with one command, a near-replica of the production system can be set up on a local computer. There's a step-by-step list of what's required to do this.
3. I built a small tool to help visualize the entire system when link updates are flowing through. While it's not ready to release, it's tremendously helpful in local development / gaining of an understanding of what's actually going on.
4. Fixed a couple bugs in the webhook job. Running on a current replica of production, ~80% of links are able to be processed and executed on by the worker. I suspect the last ~20% are due to a couple factors that haven't been considered:
   • Repositories being deleted that are in a link, and the worker not taking this into account.
   • Branches being deleted that the link was supposed to pull changes from or propose changes into.
5. Fixed an issue in which a worker handing a number of links could potentially exhaust the gthub token's rate limit. Unfortunately, this means it can sometimes take up to 2 minutes or so for a link operation to process, but I think that's an adequate target for now.

I'm nearly ready to release this thing. I'm a bit worried that once it's released, I will have forgotten to verify an edge case and I'll get an angry issue, but I think I just need to bite the bullet. My goal is to release this new stuff by next weekend.

[1egoman]

Pre-deployment

• Run tests on all projects
  • worker
  • legacy
  • server
• Try out some backstroke classic links (both on upstream, and on fork)
  • On Upstream
  • On Fork
  • Ensure that on a fork that has an issue tagged optout doesn't get a PR.
• Set up 2FA on digitalocean

Deployment checklist

• Push up all backstrokeapp repos.

• Build containers for all repos.

• Tag all with v2.0.0. Ensure package.json in all also says 2.0.0 in all projects.

• Push up all backstrokeapp/server code to 1egoman/backstroke.

• Remove backstrokeapp/server.

• Transfer 1egoman/backstroke to backstrokeapp/server.

• Delete droplet. Redeploy just to make sure we're in a clean state.

• Migrate all data from mongo into new database

• Make sure legacy works

• Update backstroke.us dns:

  • backstroke.us should point to the droplet.
  • www.backstroke.us should also point to the droplet. (redirect happens in haproxy).

Verify

• Verify all these redirect to https://backstroke.co:
  • https://backstroke.us
  • https://www.backstroke.us
  • http://backstroke.us
  • http://www.backstroke.us
  • http://www.backstroke.co
  • https://www.backstroke.co
  • http://backstroke.co
• Ensure that the new website displays at the apex of backstroke.co.
• Attempt to login. Verify that you get dropped at the dashboard in a logged-in state.
• Attempt to create a link. Use test upstream and fork. Verify that link can be created.
• Sync link manually, ensure that works.
• Push up tagged docker containers.

Take down old stuff (do this once sure that the new stuff is stable)

• Securely erase the (now) old mlab database.
• Take down the backstroke heroku dyno.
• Export data from mixpanel and delete it since its no longer being used. It's still being used.