Room for improvement on handling failures in migrations when deploying Pycon

[python]

Room for improvement on handling failures in migrations when deploying Pycon

[dpoirier]

We deployed a fix on Friday afternoon to our staging server, but we were still getting errors showing tracebacks from the previous code this morning. I verified that the code checked out on the server was the updated code, so apparently the server processes (or at least one of them) were still running the old code.

Here's a bit of the minion log from Friday:

2015-09-18 21:32:09,753 [salt.loaded.int.module.cmdmod][ERROR   ] Command '/srv/pycon/env/bin/python manage.py migrate --noinput && /srv/pycon/env/bin/python manage.py compress --force && /srv/pycon/env/bin/python manage.py collectstatic -v0 --noinput' failed with return code: 139
2015-09-18 21:32:09,757 [salt.loaded.int.module.cmdmod][ERROR   ] stderr: /bin/bash: line 1:  9661 Segmentation fault      /srv/pycon/env/bin/python manage.py migrate --noinput
2015-09-18 21:32:09,757 [salt.loaded.int.module.cmdmod][ERROR   ] retcode: 139
2015-09-18 21:32:09,760 [salt.state       ][ERROR   ] {'pid': 9660, 'retcode': 139, 'stderr': '/bin/bash: line 1:  9661 Segmentation fault      /srv/pycon/env/bin/python manage.py migrate --noinput', 'stdout': ''}

I think here's what happened:

• We updated the staging branch
• The next deploy checked out the updated code
• A later state segfaulted while trying to run migrations (I don't know why it segfaulted), so that Salt never got to the server restart step
• The next time the deploy ran, the code didn't need to be updated, so none of the rest of the steps that include restarting the server ever got run

I still don't have any wonderful ideas for fixing this kind of problem. See my previous comments in this issue.

[ewdurbin]

We now deploy the pycon site via Heroku, so this is no longer relevant.