mediamath/grim

[deprecated] a dead simple build server

Grim: Dead simple build server

Deprecated As of October 2018 grim relies on a github api call that no longer works. We will not be upgrading the daemon to fix this. Please find a different solution to your build tool needs.

Grim is the "GitHub Responder In MediaMath". We liked the acronym and awkwardly filled in the details to fit it. In short, it is a task runner that is triggered by GitHub push/pull request hooks that is intended as a much simpler and easy-to-use build server than the more modular alternatives (eg. Jenkins).

On start up, Grim will:

1. Create (or reuse) an Amazon SQS queue with the name specified in its config file as GrimQueueName
2. Detect which GitHub repositories it is configured to work with
3. Create (or reuse) an Amazon SNS topic for each repository
4. Configure each created topic to push to the Grim queue
5. Configure each repositories' AmazonSNS service to push hook updates (push, pull_request) to the topic

An example including two repositories watched by three Grimd's (one in EC2 and two MacBookPros).

Each GitHub repo can push to exactly one SNS topic. Multiple SQS queues can subscribe to one topic and multiple Grim instances can read from the same SQS queue. If a Grim instance isn't configured to respond to the repo specified in the hook it silently ignores the event.

Installation

1. Get grimd

wget https://artifactory.mediamath.com/artifactory/libs-release-global/com/mediamath/grim/grimd/[RELEASE]/grimd-[RELEASE].zip

2. Global Configuration

Grim tries to honor the conventional Unix filesystem hierarchy as much as possible. Configuration files are by default found in /etc/grim. You may override that default by specifying --config-root [some dir], more briefly -c [some dir] or by setting the GRIM_CONFIG_ROOT environment variable. Inside that directory there is expected to be a config.json that specifies the other paths used as well as global defaults. Here is an example:

{
	"GrimQueueName": "grim-queue",
	"ResultRoot": "/var/log/grim",
	"WorkspaceRoot": "/var/tmp/grim",
	"AWSRegion": "us-east-1",
	"AWSKey": "xxxx",
	"AWSSecret": "xxxx",
	"GitHubToken": "xxxx",
	"HipChatToken": "xxxx"
}

If you don't configure GrimQueueName, ResultRoot or WorkspaceRoot Grim will use default values. The AWS credentials supplied must be able to create and modify SNS topics and SQS queues.

Required GitHub token scopes

• write:repo_hook to be able to create/edit repository hooks
• repo:status to be able set commit statuses
• repo to be able to download the repo

3. Repository Configuration

In order for Grim to respond to GitHub events it needs subdirectories to be made in the configuration root. Inside those subdirectories should be a config.json and optionally a build.sh. Here is an example directory structure:

/etc/grim
/etc/grim/config.json
/etc/grim/MediaMath
/etc/grim/MediaMath/grim
/etc/grim/MediaMath/grim/config.json
/etc/grim/MediaMath/grim/build.sh

The file config.json can have an empty JSON object or have the following fields:

{
	"GitHubToken": "xxxx",
	"HipChatToken": "xxxx"
	"HipChatRoom": "xxxx",
	"PathToCloneIn": "/go/src/github.com/MediaMath/grim"
}

The GitHub and HipChat tokens will override the global ones if present. The HipChat room is optional and if present will indicate that status messages will go to that room. The field PathToCloneIn is relative to the workspace that was created for this build.

Build script location

Grim will look for a build script first in the configuration directory for the repo as build.sh and failing that in the root of the cloned repo as either .grim_build.sh or grim_build.sh.

The environment variables available to this script are documented here.

Environment Variables

CLONE_PATH= the path relative to the workspace the repo is cloned in
GH_EVENT_NAME= either 'push', 'pull_request' or '' (for manual builds)
GH_ACTION= the sub action of a pull request (eg. 'opened', 'closed', or 'reopened', 'synchronize') or blank for other event types
GH_USER_NAME= the user initiating the event
GH_OWNER= the owner part of a repo (eg. 'MediaMath')
GH_REPO= the name of a repo (eg. 'grim')
GH_TARGET= the branch that a commit was merged to
GH_REF= the ref to build
GH_STATUS_REF= the ref to set the status of
GH_URL= the GitHub URL to find the changes at

grim's Issues

Default error template uses incorrect template name

https://github.com/MediaMath/grim/blob/master/config.go#L194

No .logDir needs to be .LogDir

grim job config should allow the configuration of job specific environment variables

Should be able to set environment variables for grim builds via the job config script.

Global and local config timeouts do not seem to be honored

Add rebuild command to grimd

Should take a workspace on the server and rerun it

Will need to also store the environment that is run.

Update godoc

Overview is empty. Also should review all of the comments and prove that golint passes.

default timeout is too short

Should up it to like 5 minutes or something.

Write prepare.sh output as it happens

grim fails on startup if it has jobs configured for nonexistant repos or if it does not have access to the repo

make integration tests that require credentials easy to setup so that they are safe from leaking credentials

Should be able to setup easily a credentials file in the users home directory or something that will enable the integration tests so that they can be run safely and regularly.

Include instructions for this in README.

grimd integration tests

A lot of what grim does can only be seen when interacting with external systems. Setting up those external systems is hard. We need to add automated integration tests.

System clock drift causes git checkout problem

If the system clock drifts too far github checkouts will fail.

The provision script could include install of ntp to prevent this.

grimServerId should not be > 15 characters

One option is to truncate it and log that it was truncated with a message that outlines how to update it.

slack integration

Make grim ami public

grimd should startup even when it cant create hooks for a repo

Take a stand on how to do github clone access

Currently we leave it murky how one should set up grim to clone github repos. In our build server we use ssh private keys (and not well I might add) but that is not documented in the grim documentation. We should move to supporting and encouraging oauth based cloning (as we already require oauth access for other things and oauth access is at least as limited as private keys or deploy key, if not org machine users).

To that end:

1. document the choices and how to implement them.
2. add the oauth token as an argument to the prepare.sh script (but only to this, very dangerous to include in build scripts).
3. create a "prepare" sub command that outputs our "preferred" prepare.sh script.

Update grim to use v2 of hipchat

Current library doesn't support it. V2 appears to have better support for non admin tokens.

Build start message has weird () as workspace doesn't appear in the notification message

https://github.com/MediaMath/grim/blob/master/config.go#L189

"Starting build of MediaMath/janus initiated by a push to infra-550-add-ability-to-add-third-parties by logie17 ()"

() is weird.

Add timeout to builds

Should be configurable per repo, globally and defaulted to something reasonable.

Save old grim amis

So we dont ruin it for our users.

Include build server identification in hipchat messages

In the case of multiple grim servers (either clustered or other) it is very useful to know "which" grim server is generating a hipchat message.

We should:

1. Add a root configuration option like "grib_server_id" or something. If not provided use the existing grim_queue_name.
2. Include that configuration option in each of the hipchat messages.

sns topic name should appear in validation errors about it

https://github.com/MediaMath/grim/blob/master/localconfig.go#L72

Allow for secret storage in grim configs

We have potential jobs that need secrets to run their integration tests. Storing those secrets in the code repos is problematic (and in open source projects completely unacceptable).

If we could store them in grim configs then the grim deployment jobs (which assumedly all ready do secret storage correctly right? RIGHT?) could embed them in the configs.

dir_test has skipped test that was failing

circumstantial evidence that builds are marked notified as successful on timeout

Had a build end in the middle of a build with no error code. It was marked as successful by grim. Possibly related to timeouts.

grimd panic if config.json not present in repo.

When grimd runs, it looks for a config.json in repo, config.json is of following format:

{
    "GitHubToken": "",
    "HipChatToken":"",
    "HipChatRoom":"",
    "PathToCloneIn":"/go/src/github.com/MediaMath/test/grim"
}

If the file is not found by grimd, it panics with following error message -

panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x7413f]

Log every hipchat notification

Hipchat is not reliable, but for tracking things down it is very useful to see the hipchat flow.

logdir notification template should include the full path to the results dir

currently only goes to the repo specific and not job specific location.

Make log and workspace dirs use the same timestamp in their names

grim needs at least 1 integration test that runs from the repo without difficulty

Most of the integration tests require an amazon/hipchat/or private github key. we need to write an integration test that can be run by anyone who forks the repo at any time.

At a minimum it should trigger the "build" command. That is, clone a repo, run a build script get results etc.

Grim should briefly log a received hook and why it either builds or ignores it

Better notification/logging when no grim script is available

When a build job is triggered that cannot find a build script the following is added to the logs:

"error while performing build: fork/exec : no such file or directory"

and the following is added to hipchat:

"Error during build of MediaMath/Keryx initiated by a ..."

As this is frequently the first interaction with grim that someone has (setting up a job) it behooves us to make this a little nicer.

Should make the error more specific in both cases.

Write job output immediately instead of at the end

grim should support git submodules

Currently it uses the github http api and is not getting submodules.

SNS Topic Name improvement

SNS Topic creation should look like the following:

1. look for existing topic on repo
2. If topic exists and permissions are ok for adding to the grim queue, do so.
3. If topic exists and permissions are not ok for adding to the grim queue, error.
4. If topic does not exists, create it with a "safe" name that is not configurable.

There is potential for races in 4, so need to give that some thought.

grimd log should have build information in it

grimd logging

A) make results directory at beginning of build.
B) log the hook to results
C) log to the results directory at a minimum:

• build start
• workspace created
• build script started
• build script finished error/no error

D) grimd logging should include the creation of the results directory at a minimum.

Grim build timeout only kills top level process. Leaves any processes it starts running.

grim makefile assumes $GOPATH is a single variable

and that ain't always true.

Configurable hipchat notification colors

Config will be the hard part...

timeout logic is odd

timeout doesnt actually kill the goroutine so you can get a timeout followed by a success. further goroutines can stack up taking resources.

grim should ignore branch deletions

Write hook events to the log

grim should handle interupts "correctly"

Add log directory to notify context so it can be optionally included in templates

Currently have workspace but to actually see the errors log directory is better.

Wonder if we should combine those?

Integration tests that require authentication should be run on startup and results logged.

It is too problematic to embed credentials into any of the builds (we are accepting pull requests from the internet which trigger jobs).

Instead we should migrate the existing tests that take credentials into a health check sub command of grimd. Replaced #58

Switch install instructions to either go get or public artifactory repo

Once this is a public repo gosh is not an appropriate install instruction.

build errors are overly thought of as fatal

audit the handling of the "build" functions errors to make sure that it is handling them correctly.

Fix aws-go-sdk switch from *map to just map

This is a breaking change and off course it is breaking Grim.

/sqs_queue.go:97: cannot use map[string]*string literal (type *map[string]*string) as type map[string]*string in field value
./sqs_queue.go:151: invalid indirect of resp.Attributes (type map[string]*string)
./sqs_queue.go:168: cannot use map[string]*string literal (type *map[string]*string) as type map[string]*string in field value

need GRIM fix.

Make provisioning not based on private changes ami

If we want to make the ami for this public we should probably make the provisioning for this repo not based on a private ami.