twisted-infra/braid

Automation scripts for twistedmatrix.com

braid's Issues

Clean up module-level initialization in `braid.config`

"Command(s) not found: t-web.uploadRelease"

Following the instructions in the ReleaseProcess[1] on the wiki, I run into an error on the final step:

(braid)red ~/braid/ (master) $
cd ~/braid; fab config.production t-web.uploadRelease:13.2.0,~/release.tar.bz2

Warning: Command(s) not found:
    t-web.uploadRelease

<rest of available commands here>

I'm not sure if anything is in a branch, so if it is, the ReleaseProcess document should match it. If the functionality doesn't exist, it should be reverted to the old version.

[1] https://twistedmatrix.com/trac/wiki/ReleaseProcess

Use absolute paths in run/sudo.

froztbyte commented:

I tend to use absolute paths for system utils just in case there's PATH fuckery (a good recent example I've run into is sensible-browser vs xdg-open vs not having either of those), and would suggest perhaps just being a bit more specific about the path layout of all the various bits which braid sets up.

Properly install twisted's trac plugins.

Currently we just have eggs in version control, rather than the sources.

Generate a test-dataset for Trac

Figure out and document how to handle private config data and live data.

trac will need this, and twisted-buildbot-configuration needs this (it actually needs it during install to run buildbot upgrade-master).

Create command to upload ssl keys for twisted-web.

Create a system-admin user to administer services

By creating a specific user to administer services, and using its .ssh/authorized_keys as target for the Match Group directive, we can assign different privileges to users (install/setup for root only and restart/update for every service admin).

Functions to add/remove keys to the authorized_keys file should be provided.

buildbot fails to dump due to size.

Provide a way to admin trac remotely.

.local/bin/trac-admin config/trac-env/ permission add <username> wiki_editors

for example

Make twisted-web able to start the maintence-site automatically.

trac.start fails without manual intervention because htpasswd is missing

In braid.twisted.service, link all scripts that we know about.

Rather than having each class link ~/config/start to ~/bin/start, have the base class do this, and also do it for stop if that exists, rather than using the default one.

Also, we could perhaps generate a typical start, and we should create a restart as well.

Use procmon to make trac have a single start and single stop script.

The monitor can be a separate service in the same tac file, that just triggers trac (but not the post-commit-hook) to restart.

Put buildbot's private config somewhere private but accessible.

This probably means on wolfwood. One tricky bit, is how to make it accesible to the service user.

Either there needs to be a shared user on wolfwood for accessing it, or people need to put their user in ~/.config/braid/production.env somehow. (But the latter means the remote won't be correct).

Braid codespeed.

Consider using pam_ssh_agent_auth

Update twisted-website to get all non-versioned data from a single directory.

My notes indicate that this is currently ~/twisted/Releases and ~/distributable.

There should be commands to dump+restore that directory, as well.

add fail2ban configuration to braid

fail2ban is now installed into the dornkirk mailman configuration but its deployment is not automated. We should automate that.

Generate crontab's automaitcally, rather than having identical copies everywhere.

buildbot and trac will actually addtional things to go there (see #23).

t-web.start tries to run /opt/pypy-2.0.2/bin/twistd

but there is no such thing. t-web.install installs twisted with --user putting twistd at ~t-web/bin/twistd

trac-config needs dump/restore tasks.

The data that needs to be preserved is the db, ~/attachments and htpasswd.

Update trac's release-macro to point at the correct place.

The website has moved, and #18 is moving the subdirectory where the releases live.

Provide a better method of doing interpolation into shell commands.

Is there something as convenient as .format, but that properly handles
shell quoting?

'from pipes import quote'. You'll need to write a function that combines that with 'format', of course.

Note that there are a couple of places where we currently depend on spaces in interpolated strings causing arguments to be separated. (Nothing that isn't fixable, but something to look out for an fix, when addressing this.)

Install PyPy from PPA rather than binary tarball

Packages are nice, we should use packages.

Among other things, the current PyPy deployment situation means that we can't deploy on an i386 VM for testing.

`users.uploadKeys` makes authorized_keys root-owned, if it creates it.

twisted-buildbot need commands to dump/restore live data.

This is all the directories in ~/data plus state.sqlite there.

dependencies is a special case, but does need to be handled as well.

Execute tasks in parallel by default

Currently, tasks which support multiple target hosts, are run in sequence. It is possible to run them in parallel by passing the -P command line flag when invoking fabric.

We can make this behavior the default by wrapping the generated tasks in the parallel decorator inside Service.getTasks.

archive.restore doesn't properly handle unexpected files in the dump.

Make git/bzr check that the repository is properly set, when updating.

t.names commands fail

$ fab -f fabfile.py t-names.start

Fatal error: The following specified roles do not exist:
nameserver

Aborting.

eliminate "server dependencies" section

Currently 3 dependencies are listed:

1. the target server must have 'precise' installed
2. the target server must have 'universe' enabled
3. the target server must have 'sudo' installed.

It would be nice to eliminate this documentation and, for each of these, respectively:

1. run a command to verify that 'precise' is installed, and give a useful, comprehensible error message if it isn't.
2. enable the universe component automatically
3. install sudo automatically

Make environment shortcuts easy to integrate

Currently environment shortcuts (test and prod) have to be imported manually into all service fabfiles which want to make use of them. We should eventually get rid of them completely or make them a generic single import.

A possible solution can be to move them to the braid.env package and make the environment task the default one. This would have as consequences a slightly longer but more explicit command line syntax: fab env.test myTask

trac.install doesn't initialize trac's postgresql database at all

so trac can't actually be started just from braid setup, a trac environment needs to be copied from somewhere else or otherwise manually initialized

Create README's for all services

They should say what the service is, pointers to the config file, additional paths used, fabric commands available, dump/restore procedure, etc.

• t-names
• twisted-website
• buildbot
• diffresource
• kenaan
• hiscore
• trac
• mailman

kenaan needs to be able to handle a private config.

The current deployment uses a local branch, but it should be a seperate repo.

It probably wants to live in ~/private

Clean up generation of service README.

The code probably wants to be factored into a function for generating the readme, as well as a generic function like fabric.contrib.file.upload_template.

installation fails when trying to use pip to install things for PyPy

Both pyOpenSSL and Twisted installation on PyPy fails:

[54.242.142.31] sudo: ~pypy/bin/pip install svn+svn://svn.twistedmatrix.com/svn/Twisted/trunk/
[54.242.142.31] out: Wheel installs require setuptools >= 0.8 for dist-info support.
[54.242.142.31] out: pip's wheel support requires setuptools >= 0.8 for dist-info support.
[54.242.142.31] out: Storing debug log for failure in /home/ubuntu/.pip/pip.log
[54.242.142.31] out:

Install `.bashrc` for service users that sets PATH etc appropriately.

If using virtualenv, this means sourcing the virtualenv and ~/bin.

Otherwise ~/bin and ~/.local/bin at least (and perhaps ~pypy/bin if appropriate).

`equivs` is used but never installed

trac.install ignores username passed to fab

fab -u somebody ... trac.install still logs in to the server as root

if somebody has sudo permission that should be sufficient.

mailman needs twisted-website to run as group www-data.

mailman's wrapper (used by the cgi script, limits the groups that can access it.

can't easily test t-web on another host

t-web configuration redirects to twistedmatrix.com so it's hard to load content just from an alternate testing host

Implement a check that the repo is clean and push upstream when running in production.

Code used to deploy to production should be recorded, so we should ensure that that is being done.

Upgrade pypy install to release version.

installation on a server with out-of-date dpkg repository info fails

The braid.bootstrap step fails hard if apt-get install ... fails.

It could run apt-get update first.

Make `braid.pip.install` smarter about picking which `pip` to run.

Using virtualenv (as suggested in #1) would address this. The solution in 67ca922 works for the moment, but is a bit of a hack.

Implement buildbot and trac's cleanup cron jobs.

Use virtualenvs

(just keeping track of things to discuss over here)

Virtual environments should probably still be preferred to user local installations as they provide more isolation from the system site-packages, allow to run different versions of python and don't require scripts to be invoked by specifiying the correct interpreter.

We could also activate the correct environment in each user's .profile in order to have it available for each command.

Update kenaan config with real data.