jonasmalacofilho/robrt

I'm Robrt, a robot that listens to GitHub events and deploys stuff

robrt's Issues

Let's open source!

I'm considering open sourcing this right now, in it's current state.

Sure, there are security issues, but keeping this hidden isn't the way to deal with that. Also, someone might actually help me fix some of these issues, either to use Robrt or simply to prove a point. Not to mention that those issues really do need to be fixed before the release of the online BRTPG, so there's no way around them...

And sure, there are other things I would like to change and/or improve before calling this stable, but I see no reason why I can't do that in public. Indeed, by having it visible to everyone I'm more likely to keep working on this.

---

@aszasz I'm sure you don't have any objections, but do let me know your thoughts about this...

Also, I would like your opinion on the license: the usual BSD 2-clause (or 3-clause), the MIT license or GPL?

Failed branch builds filtered from exporting still emit Success events

[2792276d] ABORTING: non zero status  @robrt.PushBuild.build(PushBuild.hx:454)
[2792276d] notify: build-failure  @robrt.PushBuild.log(PushBuild.hx:60)
  * enqueue "build-failure"  @robrt.GitHubNotifier.notify(Notifier.hx:125)
  * immediatly popping  @robrt.GitHubNotifier.notify(Notifier.hx:128)
  * popping (source was Notifier.hx:130)  @robrt.GitHubNotifier.pop(Notifier.hx:76)
  * dequeue "build-failure"  @robrt.GitHubNotifier.pop(Notifier.hx:86)
[2792276d] build successfull  @robrt.PushBuild.build(PushBuild.hx:485)
[2792276d] branch content filtered out from exporting  @robrt.PushBuild.export(PushBuild.hx:514)
[2792276d] notify: no-export  @robrt.PushBuild.log(PushBuild.hx:60)
  * enqueue "no-export"  @robrt.GitHubNotifier.notify(Notifier.hx:125)
[2792276d] exporting the build log  @robrt.PushBuild.export(PushBuild.hx:522)
[2792276d] starting cleanup  @robrt.PushBuild.doCleanup(PushBuild.hx:563)
[2792276d] cleanup: remove the base build dir  @robrt.PushBuild.doCleanup(PushBuild.hx:565)
  * will pop again only after 500ms (event was "build-failure")  @robrt.GitHubNotifier.pop(Notifier.hx:102)
  * popping (source was Notifier.hx:103)  @robrt.GitHubNotifier.pop(Notifier.hx:76)
  * dequeue "no-export"  @robrt.GitHubNotifier.pop(Notifier.hx:86)
[2792276d] build failed with 500  @robrt.IncomingRequest.execute(IncomingRequest.hx:110)
[2792276d] Done  @robrt.IncomingRequest.handleRequest(IncomingRequest.hx:154)
  * will pop again only after 500ms (event was "no-export")  @robrt.GitHubNotifier.pop(Notifier.hx:102)
  * popping (source was Notifier.hx:103)  @robrt.GitHubNotifier.pop(Notifier.hx:76)
  * no more items in queue  @robrt.GitHubNotifier.pop(Notifier.hx:79)

Both build-failure and no-export were emitted.

Ref: http://robrt.io/?https://lbelcard.com.br/robrt/logs/2792276d.txt

Support for organization repositories

Idea: use GitHub commit statuses as a key value database

Basically, it might be possible to implement #12 (Add a database), at least in part, in a stateless server (where all state stays at GitHub).

Implement git-lfs support at the host level

Related to: ITDP/the-online-brt-planning-guide#129

Replace `node-remove` (and `mkdir-p`) by `fs-extra`

The implementation of node-remove is somewhat fragile (I could get its tests to run) and broken (see dsc/node-remove#2). I think we should replace it with fs-extra, taking the time to also replace mkdir-p as well.

• make basic externs for fs-extra
• replace all usage of node-remove
• replace all usage of mkdir-p
• complete the externs for fs-extra

Add event EBadToken

Command not found not being recognized as error

See: http://robrt.io/?https://brt2.robrt.io/logs/d880076b/build.txt (L1933–1934)

Cannot read property 'id' of null for "delete" event

{
  "ref": "refs/heads/master",
  "created": false,
  "deleted": true,
  "forced": true,
  "base_ref": null,
  "commits": [
  ],
  "head_commit": null,
  ...
}

results in

Error parsing request: Error: TypeError: Cannot read property 'id' of null  @robrt.IncomingRequest.execute(IncomingRequest.hx:56)

Error probably is caused by e.head_commit.id in IncomingRequest.hx#L97

Add Gitter notifications

There are basically two ways:

1. Post regular messages to the channel via the corresponding REST APIs

2. Post to the activity feed of the channel, using: (a) designing a robrt service for Gitter (i.e. webhooks); or (b) using the supplied "custom" service.

Using the "custom" webhook (alternative 2.b) is probably the best way to tackle this right now.

Add a database

This would serve many purposes:

• restart work, specially notifications
• shard the build load across servers
• provide some ui to view progress and past results
• rebuild pull requests when their base branch is updated

Support pull requests events that the base has changed

Apparently this means "action": "edited"...

Control caching of Docker images

First idea: add a preapre.cache_control setting to .robrt.json

This setting would control the reuse of the entire prepared image. It's argument would look something like no-cache, 3h, 24h. Or, for simpler parsing, 0, 10800000, 86400000 (time to cache in ms)...

Advantage: relatively easy to implement

Disadvantage: can't differentiate between long lived dependencies (such as basic build tools) and short lived ones (such as sibling libraries)

Enable GitHub commit statuses for non PR builds

GitHub commit statues also show on the branch overview page. It is therefore useful to add statuses for all builds, not only those that are pull requests.

Allow the build to be split into steps

This would allow each step to have it's own opportunity to export things and to generate notification events.

We thus could split the build of the online BRT guide into 'manu', 'html' and 'pdf' steps.

Non root mode

• connect to docker via TCP (or possibly socket activation)
• allow configuration of (or commands to change) ownership and permissions for all export paths

Allow Yaml configuration

Json is annoying to write by hand; Yaml, on the other hand, is pretty standard for CI tools.

If necessary, completely dropping Json support, server and/or client side, is acceptable.

Isolate sensitive tokens away from the web

Consider moving steps that require sensitive authentication credentials to a different process, thereby limiting the exposure from simpler potential abuses of the public facing service. This additional process should only support a very simple and very (easily) sanitized API to the main robrt service.

Note: this does not make any sense before a non root mode (issue #30) is added.

Additionally, only trusted users should be allowed to interfere with container creation and, if at all possible, builds (within containers) should not run as root.

Build failures on latest Haxe/development HEAD

It should be fairly straightforward to fix (though probably in hxnodejs).

src/robrt/PushBuild.hx:25: lines 25-37 : Field _transform overrides parent class with different or incomplete type
/home/travis/build/jonasmalacofilho/robrt/.haxelib/hxnodejs/git/src/js/node/stream/Transform.hx:56: characters 2-115 : Base field is defined here
src/robrt/PushBuild.hx:14: lines 14-38 : Defined in this class
src/robrt/PushBuild.hx:25: lines 25-37 : error: js.node.Buffer should be Dynamic
src/robrt/PushBuild.hx:25: lines 25-37 :  have: (Dynamic, ..., ...) -> ...
src/robrt/PushBuild.hx:25: lines 25-37 :  want: (js.node.Buffer, ..., ...) -> ...

There are also a couple of changes that generate quite a few warnings...

<in many places>: Warning : This typedef is deprecated in favor of js.lib.Error

/home/travis/build/jonasmalacofilho/robrt/.haxelib/hxnodejs/git/src/js/node/Url.hx:lines 61 and 83: characters 32-41 : Warning : This typedef is deprecated in favor of { ?slashes : Null<Bool>, ?search : Null<String>, ?query : Null<haxe.extern.EitherType<String, haxe.DynamicAccess<String>>>, ?protocol : Null<String>, ?port : Null<String>, ?pathname : Null<String>, ?path : Null<String>, ?href : Null<String>, ?hostname : Null<String>, ?host : Null<String>, ?hash : Null<String>, ?auth : Null<String> }

@aszasz can you fix these?

Complete log in travis.

---

P.S. I don't get notified for a failed build when the commit isn't mine, even if its from our scheduled weekly rebuilds. If you receive one of these emails, don't assume I did as well.

Improve the cleanup after all builds

Review the current algo and, by default, cleanup everything. At the same time, enable an options to keep most data (maybe add more granular settings).

Don't accept pings if the message signatures fails to validate

Correctly handle build commands to be executed in the background

The user might need to run some commands in the background, such as mongod.

In that case, the most obvious way to specify the command in the configuration file is with

mongod &>dev/null &

However, this breaks our build with:

bash: line 2: syntax error near unexpected token `;'
bash: line 2: `echo "Robrt: starting cmd <1>"; mongod &>/dev/null &; echo "Robrt: finished cmd <1> with status <$?>"'

It might suffice to separate the second echo from the user defined command.

Filtered out builds should not respond with "starting build"

Version: 711e43c

Log:

* POST / -> [56f5a78b]  @robrt.IncomingRequest.handleRequest(IncomingRequest.hx:149)
[56f5a78b] DELIVERY: *****  @robrt.IncomingRequest.execute(IncomingRequest.hx:61)
[56f5a78b] repository: jonasmalacofilho/*****  @robrt.IncomingRequest.execute(IncomingRequest.hx:76)
[56f5a78b] event: GitHubPush  @robrt.IncomingRequest.execute(IncomingRequest.hx:77)
[56f5a78b] repository matches: jonasmalacofilho/*****  @robrt.IncomingRequest.execute(IncomingRequest.hx:86)
[56f5a78b] action: pushed Sync  @robrt.IncomingRequest.execute(IncomingRequest.hx:103)
[56f5a78b] starting build  @robrt.PushBuild.run(PushBuild.hx:491)
[56f5a78b] notify: started  @robrt.PushBuild.log(PushBuild.hx:59)
[56f5a78b] branch filtered out from building  @robrt.PushBuild.run(PushBuild.hx:499)
[56f5a78b] notify: no-build  @robrt.PushBuild.log(PushBuild.hx:59)
[56f5a78b] Done  @robrt.IncomingRequest.handleRequest(IncomingRequest.hx:152)

Response: 202 Accepted, starting build id 56f5a78b

Replace the use of image tags for image ids

Add environment variables to the server config

Create some way of pinging the server without any specific payload

This is for automated up/down checks, like uptimerobot.

Unhandled exception when sending notifications

Sep 21 19:08:01 brt.robrt.io env[1498]: at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)
Sep 21 19:08:01 brt.robrt.io env[1498]: at errnoException (dns.js:26:10)
Sep 21 19:08:01 brt.robrt.io env[1498]: Error: getaddrinfo ENOTFOUND api.github.com api.github.com:443
Sep 21 19:08:01 brt.robrt.io env[1498]: ^
Sep 21 19:08:01 brt.robrt.io env[1498]: throw er; // Unhandled 'error' event
Sep 21 19:08:01 brt.robrt.io env[1498]: events.js:141

Make the export happen (closer to) atomically

Export optimizations

• actually make it as close as possible to atomic (possibly by combining the symlink and rename syscalls)
• try to use rename for the actual data export, instead of copying all files (but handle the limitations of rename)

This is in part a continuation of #17.

Comprehensive and realtime logs

• complete pull request #20 ("Stream the build log in 'real time'")
• implement log viewer issue #1 ("Add live view for streaming logs")
• add simplified and/or sanitized prepare and export steps to the log