Automated push to deployment via GitHub PRs

[jupyterhub]

Automated push to deployment via GitHub PRs

[yuvipanda]

@willingc I think we should just do staging and prod now, and not do any traffic redirection yet. +1 on having a specific 'you have deployment power, now here are your responsibilities' doc. For rolling back, we just revert the PR and merge that - I think that's sufficient for now.

[yuvipanda]

Some common tasks we should have a good documented workflow for:

1. Just changing the config items in this repository & pushing it out
2. Making a change in the http://github.com/jupyterhub/binderhub repository & then pushing that out without any config changes.
3. Making a change in the binderhub repo + related changes in this repo & pushing them out atomically.

A PR should be our atomic unit of deployment / rollback.

[yuvipanda]

Now deployment won't mark as succeeded until:

1. All pods have finished creating (5min timeout)
2. An end to end test that tries to launch a known good github repo onto binder succeeds.

This should give us more confidence in our deployments!

I'm going to close this issue now, since we've separate issues for more specific things we wanna do.

\o/

[yuvipanda]

/cc @minrk @willingc @choldgraf objections to this plan?

[choldgraf]

Agree - we'll need to document that workflow for the team but it sounds like a net positive to me for sure. If it works well for datahub then I think it's worth trying here.

[minrk]

I think this is a great plan. For details, I assume that 'staging' is actually the 'master' branch, and then we push to 'prod'? This might also be a case for trying out protected branches on this repo.

[willingc]

Big +1. No objections as this worked well for other Django projects that I've worked on (whether automated through travis or deployment scripts) when master -> staging -> prod.

A few items to think about:

• Do we use master, staging, prod or just master/staging, prod? I guess it depends if we wish to do some load testing on staging (or if staging serves a small sample of actual users) before release to prod?

• It is the responsibility of the human who merged the PR to make sure the deployment succeeds, and if it doesn't, debug it + roll back if necessary.

  • I would expand to the responsibility of 1) the human who merged and 2) the author of the PR.
• I'm assuming that we will have a bot of some sort to do rollbacks easily and consistently.

[yuvipanda]

@minrk For Berkeley (and in general) I prefer calling the branches 'staging' and 'prod', and have them correspond throughout - to cluster names, namespaces, release names, domain names, etc. I'd like to do the same here. +1 for using protected branches - we've those too!

[yuvipanda]

A very big chunk of this is done! There's some preliminary documentation at https://github.com/jupyterhub/mybinder.org-deploy#deploying-a-change. Here's an example of deploying something to staging: #15. And then deploying that to beta: #16.

I'm unsure how to get rid of the merge commits, unfortunately. 'rebase and merge' on the github UI still changes the hash, so not sure what to do here.

I've opened #11 separately for writing up deployment guidelines & responsibilities.

[yuvipanda]

I've also opened up #10 which should wait until jupyterhub/binderhub#135

[minrk]

I'm unsure how to get rid of the merge commits, unfortunately.

What would be the goal of discarding the merges? It seems like preserving them would show a trail of what's been deployed to prod vs staging, which would otherwise be discarded.

[yuvipanda]

@minrk I guess it's ok. I was only slightly annoyed at them because it means beta and staging branches will never actually be the same (since there will always be a merge commit on top in beta) but I guess that's ok.