raku/whateverable

🤖 Different IRC bots that operate on a bunch of prebuilt Rakudo versions

Whateverable

The Whateverables are a collection of IRC bots primarily useful for Raku developers. They are written in Raku and are based on IRC::Client. Many of the use cases involve running Raku code using pre-built versions of the Rakudo compiler for each commit.

Usage

See wiki for more information.

Contributing

See CONTRIBUTING.md.

Installation

See wiki/installation.

whateverable's Issues

Build rakudo on all available tags

So that one can use 2013.01.

Sure, bisectable is not going to give very meaningful results in such cases, but it will be good enough (basically it will skip everything besides tags, and then will give you the list of all commits between two tags – that should be pretty useful anyway).

Make it possible to bisect nqp and MoarVM

Not sure how this should be implemented exactly, but it would be nice to have.

segfaults and other stability issues

See this: http://irclog.perlgeek.de/perl6/2016-08-19#i_13055233

Pretty sure that it is not our fault, but we have to rakudobug it.

Is benchable broken?

See this: http://irclog.perlgeek.de/perl6/2016-09-22#i_13255837

More ways to trigger evalable

<seatek> say '123abcg' ~~ /<xdigit + [g]>+/
<seatek> m:say '123abcg' ~~ /<xdigit + [g]>+/
<seatek> m: say '123abcg' ~~ /<xdigit + [g]>+/
<camelia> rakudo-moar 906719: OUTPUT«5===SORRY!5=== Error while compiling <tmp>␤Quantifier quantifies nothing␤at <tmp>:1␤------> 3say '123abcg' ~~ /<xdigit +7⏏5 [g]>+/␤»

m: without a space should work. Also, perhaps it should be able to figure out that say …something… should be executed as well. How often people start their sentences with “say” for other reasons? If there are any false positives, perhaps also check if the message has too many symbols or something.

Make it possible to get stdout separately from stderr

https://irclog.perlgeek.de/perl6/2016-10-21#i_13439518

Add Evalable

Evalable is basically Committable that defaults to HEAD. Just find a way to subclass it.

Eval stuff securely

This is going to be easy once the work on #23 is finished.

‘help’ command should point to the wiki

We have a bunch of useful stuff there: https://github.com/perl6/whateverable/wiki

Implement some sort of a cache

Most queries are using HEAD and “releases”, there is no need to extract these files each time (wasting 0.2s on each archive each time).

In fact, it looks like even Benchable is not smart enough to not work with archives all the time (see this). Some kind of a global cache will eliminate the need to fix Benchable.

bisectable: show actual starting points somewhere in the output

Otherwise people don't really see what is being bisected.

Is there any way to do that without printing a fourth line?

Build any dangling branches

Right now we only build 2015.10..HEAD. This means that any of the branches are not being built (they are not part of that range).

But it would be nice if people were able to run stuff on experimental branches. This would also reduce the amount of time required to keep up with HEAD when branches with lots of commits are being merged.

Do not use SIGHUP to stop processes

See this TODO from Whateverable.pm6:

    if not $promise.status ~~ Kept { # timed out
        $proc.kill; # TODO sends HUP, but should kill the process tree instead
        $out.send: “«timed out after $timeout seconds, output»: ”;
    }

If I recall correctly, this prevents us from running the bots under nohup (which is not required, but still).

Create statisfiable

@timo

[13:08] <AlexDaniel> MasterDuke: timo came up with an idea for a bot that can gather some stats about startup ram usage, rakudo setting file size, etc.
[13:08] <AlexDaniel> what would be the best way to call it?
[13:09] <AlexDaniel> timo said “statisfiable”, which is probably alright
[13:09] <AlexDaniel> but who knows, maybe you have a better idea
[13:11] <timotimo> yo
[13:11] <timotimo> MasterDuke: i had some ideas for new functionality in The Whateverable Cloud
[13:11] <AlexDaniel> m: say ‘hi’
[13:11] <evalable6> AlexDaniel, rakudo-moar b5aa3c5: OUTPUT«hi»
[13:12] <timotimo> i'd like to collect the maxresidentk from perl6 -e '' over all our collected builds
[13:12] <timotimo> also maybe the maxresidentk of echo "" | perl6 for how good our repl conserves memory
[13:12] <AlexDaniel> perhaps “infoable” is ok
[13:13] <timotimo> i'd like "statisfiable" ;(
[13:13] <AlexDaniel> ah well, if you like then go with it
[13:13] <timotimo> easiest thing to bikeshed, though
[13:13] <timotimo> ... botshed?
[13:14] <timotimo> well, that and the size of our CORE.setting.moar files
[13:14] <AlexDaniel> all these bots allow people to misspell their names, so it doesn't matter as much :)
[13:14] <timotimo> hah, oh jesus :)
[13:14] <AlexDaniel> bsectble6: help
[13:14] <bisectable6> AlexDaniel, Like this: bisectable6: old=2015.12 new=HEAD exit 1 if (^∞).grep({ last })[5] // 0 == 4 # RT128181
[13:15] <AlexDaniel> blevalable: say 42
[13:15] <evalable6> AlexDaniel, rakudo-moar b5aa3c5: OUTPUT«42»
[13:15] <timotimo> besctabel: help
[13:15] <AlexDaniel> well, not that much
[13:15] <AlexDaniel> ibesctabel6: help
[13:15] <AlexDaniel> ibsctabel6: help
[13:15] <AlexDaniel> ibsctable6: help
[13:15] <bisectable6> AlexDaniel, Like this: bisectable6: old=2015.12 new=HEAD exit 1 if (^∞).grep({ last })[5] // 0 == 4 # RT128181
[13:20] <timotimo> so maybe MasterDuke has some additional ideas to what kind of stats we should collect
[13:20] <timotimo> and make available
[13:21] <AlexDaniel> what's the current list?
[13:21] <AlexDaniel> memory, CORE.setting size, startup time?
[13:23] <timotimo> startup time is not helpful, i don't think
[13:23] <timotimo> unless we time it with callgrind and get the raw Ir values

make it possible to set STDIN of all the bots

Make it possible to test stuff with 32-bit rakudo

<timotimo> just fire up a qemu :)

Easy to say, right? :)

Anyway, we should probably do it.

When setting custom stdin it probably should not allow perl mimetype

This line is used for both fetching the code and data for stdin:

if not $response.content-type.contains(any(‘text/plain’, ‘perl’)) {

text/plain I understand, but perl is not related in this case.

Add repeat=N option to make it easier to bisect flappers

Sometimes wrapping your code with for ^20 { } is not enough.

Generate @releases array automatically

See this.

Currently @releases array is hardcoded. It should not be this way.

Sometimes bots return “https://gist.github.com/” as a link

See this log: https://irclog.perlgeek.de/perl6/2016-11-09#i_13540054
However, it is not reproducible! See this: https://irclog.perlgeek.de/perl6/2016-11-10#i_13543763

Refuses to take a text/plain content, expecting text/plain; charset=utf-8

<Zoffix> committable, 5605d5f http://temp.perl6.party/foo.txt
<committable> Zoffix: It looks like a URL, but mime type is 'text/plain' while I was expecting 'text/plain; charset=utf-8'. I can only understand raw links, sorry.

Add timeouts

Given that these bots were initially made for developers, there are no timeouts whatsoever. If something takes 200 seconds to run then nothing will kill it.

Sometimes that's fine, but given that we now use these bots for various random stuff, it's time to implement a time limit. Would be great if it was configurable per bot. This way, for example, we can have 200s timeout for bisectable, 20s timeout for evalable, 50s for committable and so on.

Benchable should error out before it starts doing anything

<AlexDaniel> benchable6: say (5..Inf).reverse.list # well, let's bench it
<benchable6> AlexDaniel, starting to benchmark the 1 given commits
<benchable6> AlexDaniel, ¦«say»:Cannot find this revision

First message is not needed here.

Colorful output

Right now we turn off the colors. Not sure how to have colors in github gists, but we should at least do something for IRC output.

It is a bit tricky because before we get the actual output we can't know whether we want it in color or not. Still, it is doable.

Build Rakudo on JVM too

It would be nice to have it.

bisectable says “'bisect run' failure” when there is more than one commit to report

See this: https://gist.github.com/Whateverable/bff288f53af83bbb02d666e581a6ce68

Git output from the log:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
e609822f01b82608b1b1c869032c95e9641172ce
615d30c39eba318f7449b94bbbd0295e2fd75387
8beb87b415014254412409d56378afb3ed5f14d9
05170e0c14969663c816ee1c5aaa019d70938247
899e0fd4c1ce51bc492666ffc69627c502f657f7
e5443765d4bce0697c6191dd2b17db3044e027ab
fcd0093b43614ce91caeb8c23e2bbcff68d54f72
04929feeb06fa851d77ef5efa6be35248301f88c
7ee6578ce99aae069349c283012ae7e45dfd75ec
519e76487ddd1546b319bfbd47661beac2417d61
We cannot bisect more!
bisect run cannot continue any more

↑ this is a meaningful result, but bisectable says:

<bisectable> AlexDaniel: 'bisect run' failure

That's not really a failure.

When setting custom stdin it should not attempt to reply back with a full string

Take a look at this log:

<AlexDaniel> committable6: stdin http://websitetips.com/art​icles/copy/lorem/ipsum.txt
<committable6> AlexDaniel, Successfully fetched the code from the provided URL.
<committable6> AlexDaniel, https://gist.github.com/15e3​32d3fd7bb9c084b62cdbd2103fd0

It should not reply with a link to github. In this case it is probably enough if it just says “OK, custom STDIN is set” or something like that.

Committable crash

See this: https://irclog.perlgeek.de/perl6/2016-11-10#i_13546322

Here is stdout:

Unhandled exception in code scheduled on thread 19
Unhandled exception in code scheduled on thread 11
Cannot send a message on a closed channel
  in block  at /home/bisectable/git/whateverable/Whateverable.pm6 (Whateverable) line 95
  in any  at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1
  in sub THROW at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1

Cannot send a message on a closed channel
  in block  at /home/bisectable/git/whateverable/Whateverable.pm6 (Whateverable) line 96
  in any  at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1
  in sub THROW at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1

Make sure that exit code on old revision is not 125

Because 125 is “skip”. It may break the logic in some places.

Change everywhere: “good” → “old”, “bad” → “new”

Otherwise people are getting confused.

Git already treats these as synonyms.

Show segfaults

For example, see this: https://gist.github.com/Whateverable/9f52f2a53e11a4d9519e5cf203e3b714

It segfaults in the end, but it is not displayed.

Bisectable probably should not show exit code if something crashed with SEGV

https://irclog.perlgeek.de/perl6/2016-11-13#i_13560858

Run multiple pieces of code in benchable

See http://irclog.perlgeek.de/perl6-dev/2016-08-04#i_12967844

It would be nice to be able to run difference pieces of code and compare the timings (e.g., raw numbers, how much faster/slower one is over the other).

Bisectable does not work well with segfaults, committable is OK

See this: http://irclog.perlgeek.de/perl6-dev/2016-07-28#i_12928310

We need tests

Perhaps use whatever IRC::Client is using.

Besides that, the only thing that we should mock or something is gist uploading.

Why? Because I am getting tired of testing everything manually.

• Bisectable
• Committable
• Benchable
• Evalable

MoarVM panic

Query:
unicodable: { say ‘a’ }

Result:
MoarVM panic: Internal error: zeroed target thread ID in work pass

I am pretty sure it is not our fault, but we have to narrow it down for the rakudobug.

6lang rewrite

Here is a little TODO list to track the progress:

• Base class (see Whateverable.pm6)
• Committable (see Committable.p6)
• Bisectable
• Benchable
• Port commit b19c8e2 to 6lang version

Given that these are separate files, I don't think that there is a need to keep it in a separate branch. The repo may look a bit messy, but we still run Perl 5 versions, so these are still relevant.

Remove temp file name from error output

E.g., Camelia returns this:
rakudo-moar f7770e: OUTPUT«not ok 1 - ␤␤# Failed test at <tmp> line 1␤# maximum relative tolerance: 1e-15␤# actual relative difference: 1␤»

while Committable6 returns this:
iBakeCake, ¦«2016.10»: not ok 1 - ␤␤# Failed test at /tmp/ALIDT91faL line 1␤# maximum relative tolerance: 1e-15␤# actual relative difference: 1 «exit code = 1»

You could hang committable and benchable for a really long time by sleeping for the max timeout for the max number of commits

It would be good to have a timer around the loop that runs the code for each commit so we can bail out if the total time is taking too long, in addition to the alarm-based timeout for each individual run.

Make bots able to join other channels on command

Requested by @zoffixznet. See http://irclog.perlgeek.de/perl6-dev/2016-07-30#i_12939167

What is HEAD?

10:56:12 <bartolin> committable6: 2016.09,HEAD say (my int $ = 2**32) * (my int $ = 2**63)
10:56:13 <committable6> bartolin, ¦«2016.09»: 0␤¦«HEAD»: This type cannot unbox to a native integer: P6opaque, Failure␤  in block <unit> at /tmp/5gdU3BwADu line 1␤ «exit code = 1»

It says ¦«HEAD» but gives no information on how old it actually is. Should we print something like ¦«HEAD(ffeeaa22)» ?

Allow comma to separate custom good and bad revisions

See this: http://irclog.perlgeek.de/perl6/2016-09-09#i_13180322

It did not understand the query because there is a comma. It does say which starting points it is using, so it is transparent enough and in theory the user should be able to spot the issue. But there is no reason we cannot accept a comma as well.

Proper README is needed

Let's use wiki pages instead.

Support multifile gists

Not sure what would this mean exactly, but that would be great.

Create sandboxable

Wouldn't it be cool if you could install modules and test them right on one of the bots?

I propose modulable! The bot will be running in a “dirty” environment where everyone can install modules and do other stuff (like create files).

Optionally, we can preinstall all modules beforehand. Not sure if that will be viable once ecosystem becomes huge.

I don't think this idea fits well into committable/bisectable/evalable, therefore a new bot.

Depends on #25

Use pzstd instead of zstd

Should speed things up a little bit.

Allow mixed case ‘old=’/‘new=’

<timotimo> bisectable6: GOOD=head BAD=2016.08 use NativeCall; sub printf(Str $foo) is native(Str); printf("hello");
<timotimo> :\
<timotimo> what is wrong with the good= here?

Indeed, nothing. But it only understands “good=” and “bad=” in lowercase :(

Make sure total timeout is less than ping timeout or parallelize bots

According to http://irclog.perlgeek.de/perl6/2016-08-30#i_13119980, if a bot is working for more than ~250 seconds it could get disconnected from a channel. We need to make sure that bots will stop any work before they hit the ping timeout, or parallelize them so they can respond to pings while still doing real work.

Let's allow typo-ed nicknames

<jnthn> commitable: 2016.08 my @a[Bool]; dd @a
<jnthn> commitable6: 2016.08 my @a[Bool]; dd @a
<jnthn> committable6: help?
<committable6> jnthn, Like this: committable6: f583f22,HEAD say ‘hello’; say ‘world’

It could have figured it out.

Save space by compressing at least some of the builds

OK, so here is the math!

Each build uncompressed is ≈28 MB. This does not include unnecessary source files or anything, so it does not look like we can go lower than that (unless we use compression).

So how many builds do we want to keep? Well, one year from now back is about 3000 commits. This gives roughly 84 GB. And this is just one year of builds for just MoarVM backend. In about 10 years we will slowly start approaching the 1 TB mark. Multiply it by the number of backends we want to track.

Is it a lot? Well, no, but for folks who have an SSD (me) this might be a problem.

Given that people commit stuff at a slightly faster pace than the space becomes significantly cheaper, I think that we should compress it anyway (even if it is moved to a server with more space). It is a good idea in a long run. And it will make it easier for us to throw in some extra stuff (JVM builds, or maybe even 32-bit builds or something? You never know).

OK, so what can we do?

Filesystem compression

The most obvious one is to use compression in btrfs. The problem is that it is applied for each file individually, so we are not going to save anything across many builds. Also, it is only viable if you already have btrfs, so it looks like it is not the best option.

Compress each build individually

While it may sound like a great idea to compress all builds together, it does not work that well in practice. Well, it does, but keep reading.

The best compression I got is with 7z. Each build is ≈4 MB (≈28 MB uncompressed, therefore ≈7 times space saving!)

Compressing each build individually is also good for things like local bisect. That is, we can make these archives publicly available, and then write a script that will pull these archives for your local git bisect. How cool is that! That will be about 40 MB of stuff to download per git bisect, and you cannot really compress it any further anyway because you don't know ahead of time which files you would need.

This gives us about ≈120 GB per 10 years. Good enough, I like it.

Is there anything that performs better than 7z? Well, yes:

27M     c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar
9.0M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lz4
7.0M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.zip
6.7M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.gz
6.7M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lzfse
6.4M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.bz2
5.7M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7-L15.tar.zst
4.9M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7-L22.tar.zst
4.7M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lzham
4.5M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.brotli
4.3M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lrz
4.1M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.7z
4.1M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tpxz
4.0M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.xz
3.8M    c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7-extremely-slow.tar.lrz

xz is much slower during compression and is tiny bit slower when decompressing, so the win is insignificant. lrz with extreme options is much slower at everything, so forget it.

Let's compress everything together!

For the tests, I took only 7 builds:

c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7
3878066a953195276ef99739f157d38793395d06
ef04e1e07f1de0d4eb2666985c7290f96c912be6
eea786e2a2febef0ab0bfabca956beae95ab81fd
76be77c9d6e697c26e92dc704109b7b8780845aa
d30806bd00dee201cd891550ac65f084f18e8285
9bfbab9186d710e0603b1eb86be1e5ba2e0c84d1

Now, there are some funny entries here. Obviously, .tar is the size of all repos together, uncompressed. git-repo is a git repo with every build committed one after another in the right order. Look, it performs better than some of the other things! Amazing! And even better if you compress it afterwards. Who would've thought, right?

187M    all.tar
63M     all.tar.lz4
49M     all.zip
47M     all.tar.gz
45M     all.tar.bz2
42M     git-repo.tar
35M     all.tar.zstd # -19
28M     all.tar.xz
28M     all.7z
19M     git-repo.7z
9.5M    all.tar.zstd # -19 --long
8.3M    all.tar.lrz
7.3M    all.tar.zstd # --ultra -22 --long=31 --format=lzma # but it's much slower than lrz when compressing

However, lrz clearly wins this time. And that's with default settings! Wow.

Conclusion

I think that there are ways to fiddle with options to get even better results. Suggestions are welcome!

However, at this point it looks like the best way is to use 7z to compress each build individually.
Messing around with one huge archive is probably not worth the savings. It will make decompression time significantly slower, but we want decompression to be as fast as possible.