raku / whateverable Goto Github PK
View Code? Open in Web Editor NEWđ€ Different IRC bots that operate on a bunch of prebuilt Rakudo versions
Home Page: https://gist.github.com/Whateverable
License: GNU Affero General Public License v3.0
đ€ Different IRC bots that operate on a bunch of prebuilt Rakudo versions
Home Page: https://gist.github.com/Whateverable
License: GNU Affero General Public License v3.0
See this: http://irclog.perlgeek.de/perl6/2016-08-19#i_13055233
Pretty sure that it is not our fault, but we have to rakudobug it.
Take a look at this log:
<AlexDaniel> committable6: stdin http://websitetips.com/artâicles/copy/lorem/ipsum.txt
<committable6> AlexDaniel, Successfully fetched the code from the provided URL.
<committable6> AlexDaniel, https://gist.github.com/15e3â32d3fd7bb9c084b62cdbd2103fd0
It should not reply with a link to github. In this case it is probably enough if it just says âOK, custom STDIN is setâ or something like that.
According to http://irclog.perlgeek.de/perl6/2016-08-30#i_13119980, if a bot is working for more than ~250 seconds it could get disconnected from a channel. We need to make sure that bots will stop any work before they hit the ping timeout, or parallelize them so they can respond to pings while still doing real work.
<AlexDaniel> benchable6: say (5..Inf).reverse.list # well, let's bench it
<benchable6> AlexDaniel, starting to benchmark the 1 given commits
<benchable6> AlexDaniel, Š«say»:Cannot find this revision
First message is not needed here.
Wouldn't it be cool if you could install modules and test them right on one of the bots?
I propose modulable
! The bot will be running in a âdirtyâ environment where everyone can install modules and do other stuff (like create files).
Optionally, we can preinstall all modules beforehand. Not sure if that will be viable once ecosystem becomes huge.
I don't think this idea fits well into committable/bisectable/evalable, therefore a new bot.
Depends on #25
<timotimo> just fire up a qemu :)
Easy to say, right? :)
Anyway, we should probably do it.
10:56:12 <bartolin> committable6: 2016.09,HEAD say (my int $ = 2**32) * (my int $ = 2**63)
10:56:13 <committable6> bartolin, Š«2016.09»: 0â€ÂŠÂ«HEAD»: This type cannot unbox to a native integer: P6opaque, Failure†in block <unit> at /tmp/5gdU3BwADu line 1†«exit code = 1»
It says Š«HEAD»
but gives no information on how old it actually is. Should we print something like Š«HEAD(ffeeaa22)»
?
See this.
Currently @releases
array is hardcoded. It should not be this way.
Requested by @zoffixznet. See http://irclog.perlgeek.de/perl6-dev/2016-07-30#i_12939167
See this: https://gist.github.com/Whateverable/bff288f53af83bbb02d666e581a6ce68
Git output from the log:
There are only 'skip'ped commits left to test.
The first bad commit could be any of:
e609822f01b82608b1b1c869032c95e9641172ce
615d30c39eba318f7449b94bbbd0295e2fd75387
8beb87b415014254412409d56378afb3ed5f14d9
05170e0c14969663c816ee1c5aaa019d70938247
899e0fd4c1ce51bc492666ffc69627c502f657f7
e5443765d4bce0697c6191dd2b17db3044e027ab
fcd0093b43614ce91caeb8c23e2bbcff68d54f72
04929feeb06fa851d77ef5efa6be35248301f88c
7ee6578ce99aae069349c283012ae7e45dfd75ec
519e76487ddd1546b319bfbd47661beac2417d61
We cannot bisect more!
bisect run cannot continue any more
â this is a meaningful result, but bisectable says:
<bisectable> AlexDaniel: 'bisect run' failure
That's not really a failure.
Let's use wiki pages instead.
Otherwise people don't really see what is being bisected.
Is there any way to do that without printing a fourth line?
<Zoffix> committable, 5605d5f http://temp.perl6.party/foo.txt
<committable> Zoffix: It looks like a URL, but mime type is 'text/plain' while I was expecting 'text/plain; charset=utf-8'. I can only understand raw links, sorry.
See this: http://irclog.perlgeek.de/perl6/2016-09-09#i_13180322
It did not understand the query because there is a comma. It does say which starting points it is using, so it is transparent enough and in theory the user should be able to spot the issue. But there is no reason we cannot accept a comma as well.
This is going to be easy once the work on #23 is finished.
It would be nice to have it.
Perhaps use whatever IRC::Client is using.
Besides that, the only thing that we should mock or something is gist uploading.
Why? Because I am getting tired of testing everything manually.
Should speed things up a little bit.
Not sure how this should be implemented exactly, but it would be nice to have.
See this TODO
from Whateverable.pm6:
if not $promise.status ~~ Kept { # timed out
$proc.kill; # TODO sends HUP, but should kill the process tree instead
$out.send: â«timed out after $timeout seconds, output»: â;
}
If I recall correctly, this prevents us from running the bots under nohup
(which is not required, but still).
See this log: https://irclog.perlgeek.de/perl6/2016-11-09#i_13540054
However, it is not reproducible! See this: https://irclog.perlgeek.de/perl6/2016-11-10#i_13543763
Right now we turn off the colors. Not sure how to have colors in github gists, but we should at least do something for IRC output.
It is a bit tricky because before we get the actual output we can't know whether we want it in color or not. Still, it is doable.
See http://irclog.perlgeek.de/perl6-dev/2016-08-04#i_12967844
It would be nice to be able to run difference pieces of code and compare the timings (e.g., raw numbers, how much faster/slower one is over the other).
Not sure what would this mean exactly, but that would be great.
Here is a little TODO list to track the progress:
Given that these are separate files, I don't think that there is a need to keep it in a separate branch. The repo may look a bit messy, but we still run Perl 5 versions, so these are still relevant.
Because 125 is âskipâ. It may break the logic in some places.
We have a bunch of useful stuff there: https://github.com/perl6/whateverable/wiki
Sometimes wrapping your code with for ^20 { }
is not enough.
Evalable is basically Committable that defaults to HEAD. Just find a way to subclass it.
<jnthn> commitable: 2016.08 my @a[Bool]; dd @a
<jnthn> commitable6: 2016.08 my @a[Bool]; dd @a
<jnthn> committable6: help?
<committable6> jnthn, Like this: committable6: f583f22,HEAD say âhelloâ; say âworldâ
It could have figured it out.
It would be good to have a timer around the loop that runs the code for each commit so we can bail out if the total time is taking too long, in addition to the alarm-based timeout for each individual run.
<timotimo> bisectable6: GOOD=head BAD=2016.08 use NativeCall; sub printf(Str $foo) is native(Str); printf("hello");
<timotimo> :\
<timotimo> what is wrong with the good= here?
Indeed, nothing. But it only understands âgood=â and âbad=â in lowercase :(
Otherwise people are getting confused.
Git already treats these as synonyms.
See this: https://irclog.perlgeek.de/perl6/2016-11-10#i_13546322
Here is stdout:
Unhandled exception in code scheduled on thread 19
Unhandled exception in code scheduled on thread 11
Cannot send a message on a closed channel
in block at /home/bisectable/git/whateverable/Whateverable.pm6 (Whateverable) line 95
in any at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1
in sub THROW at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1
Cannot send a message on a closed channel
in block at /home/bisectable/git/whateverable/Whateverable.pm6 (Whateverable) line 96
in any at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1
in sub THROW at /home/bisectable/.rakudobrew/moar-nom/install/share/perl6/runtime/CORE.setting.moarvm line 1
Right now we only build 2015.10..HEAD
. This means that any of the branches are not being built (they are not part of that range).
But it would be nice if people were able to run stuff on experimental branches. This would also reduce the amount of time required to keep up with HEAD when branches with lots of commits are being merged.
For example, see this: https://gist.github.com/Whateverable/9f52f2a53e11a4d9519e5cf203e3b714
It segfaults in the end, but it is not displayed.
Most queries are using HEAD and âreleasesâ, there is no need to extract these files each time (wasting 0.2s on each archive each time).
In fact, it looks like even Benchable is not smart enough to not work with archives all the time (see this). Some kind of a global cache will eliminate the need to fix Benchable.
Given that these bots were initially made for developers, there are no timeouts whatsoever. If something takes 200 seconds to run then nothing will kill it.
Sometimes that's fine, but given that we now use these bots for various random stuff, it's time to implement a time limit. Would be great if it was configurable per bot. This way, for example, we can have 200s timeout for bisectable, 20s timeout for evalable, 50s for committable and so on.
So that one can use 2013.01
.
Sure, bisectable is not going to give very meaningful results in such cases, but it will be good enough (basically it will skip everything besides tags, and then will give you the list of all commits between two tags â that should be pretty useful anyway).
This line is used for both fetching the code and data for stdin:
if not $response.content-type.contains(any(âtext/plainâ, âperlâ)) {
text/plain
I understand, but perl
is not related in this case.
E.g., Camelia returns this:
rakudo-moar f7770e: OUTPUT«not ok 1 - â€â€# Failed test at <tmp> line 1â€# maximum relative tolerance: 1e-15â€# actual relative difference: 1â€Â»
while Committable6 returns this:
iBakeCake, Š«2016.10»: not ok 1 - â€â€# Failed test at /tmp/ALIDT91faL line 1â€# maximum relative tolerance: 1e-15â€# actual relative difference: 1 «exit code = 1»
<seatek> say '123abcg' ~~ /<xdigit + [g]>+/
<seatek> m:say '123abcg' ~~ /<xdigit + [g]>+/
<seatek> m: say '123abcg' ~~ /<xdigit + [g]>+/
<camelia> rakudo-moar 906719: OUTPUT«5===SORRY!5=== Error while compiling <tmp>â€Quantifier quantifies nothingâ€at <tmp>:1â€------> 3say '123abcg' ~~ /<xdigit +7â5 [g]>+/â€Â»
m:
without a space should work. Also, perhaps it should be able to figure out that say âŠsomethingâŠ
should be executed as well. How often people start their sentences with âsayâ for other reasons? If there are any false positives, perhaps also check if the message has too many symbols or something.
Query:
unicodable: { say âaâ }
Result:
MoarVM panic: Internal error: zeroed target thread ID in work pass
I am pretty sure it is not our fault, but we have to narrow it down for the rakudobug.
[13:08] <AlexDaniel> MasterDuke: timo came up with an idea for a bot that can gather some stats about startup ram usage, rakudo setting file size, etc.
[13:08] <AlexDaniel> what would be the best way to call it?
[13:09] <AlexDaniel> timo said âstatisfiableâ, which is probably alright
[13:09] <AlexDaniel> but who knows, maybe you have a better idea
[13:11] <timotimo> yo
[13:11] <timotimo> MasterDuke: i had some ideas for new functionality in The Whateverable Cloud
[13:11] <AlexDaniel> m: say âhiâ
[13:11] <evalable6> AlexDaniel, rakudo-moar b5aa3c5: OUTPUT«hi»
[13:12] <timotimo> i'd like to collect the maxresidentk from perl6 -e '' over all our collected builds
[13:12] <timotimo> also maybe the maxresidentk of echo "" | perl6 for how good our repl conserves memory
[13:12] <AlexDaniel> perhaps âinfoableâ is ok
[13:13] <timotimo> i'd like "statisfiable" ;(
[13:13] <AlexDaniel> ah well, if you like then go with it
[13:13] <timotimo> easiest thing to bikeshed, though
[13:13] <timotimo> ... botshed?
[13:14] <timotimo> well, that and the size of our CORE.setting.moar files
[13:14] <AlexDaniel> all these bots allow people to misspell their names, so it doesn't matter as much :)
[13:14] <timotimo> hah, oh jesus :)
[13:14] <AlexDaniel> bsectble6: help
[13:14] <bisectable6> AlexDaniel, Like this: bisectable6: old=2015.12 new=HEAD exit 1 if (^â).grep({ last })[5] // 0 == 4 # RT128181
[13:15] <AlexDaniel> blevalable: say 42
[13:15] <evalable6> AlexDaniel, rakudo-moar b5aa3c5: OUTPUT«42»
[13:15] <timotimo> besctabel: help
[13:15] <AlexDaniel> well, not that much
[13:15] <AlexDaniel> ibesctabel6: help
[13:15] <AlexDaniel> ibsctabel6: help
[13:15] <AlexDaniel> ibsctable6: help
[13:15] <bisectable6> AlexDaniel, Like this: bisectable6: old=2015.12 new=HEAD exit 1 if (^â).grep({ last })[5] // 0 == 4 # RT128181
[13:20] <timotimo> so maybe MasterDuke has some additional ideas to what kind of stats we should collect
[13:20] <timotimo> and make available
[13:21] <AlexDaniel> what's the current list?
[13:21] <AlexDaniel> memory, CORE.setting size, startup time?
[13:23] <timotimo> startup time is not helpful, i don't think
[13:23] <timotimo> unless we time it with callgrind and get the raw Ir values
OK, so here is the math!
Each build uncompressed is â28 MB. This does not include unnecessary source files or anything, so it does not look like we can go lower than that (unless we use compression).
So how many builds do we want to keep? Well, one year from now back is about 3000 commits. This gives roughly 84 GB
. And this is just one year of builds for just MoarVM backend. In about 10 years we will slowly start approaching the 1 TB mark. Multiply it by the number of backends we want to track.
Is it a lot? Well, no, but for folks who have an SSD (me) this might be a problem.
Given that people commit stuff at a slightly faster pace than the space becomes significantly cheaper, I think that we should compress it anyway (even if it is moved to a server with more space). It is a good idea in a long run. And it will make it easier for us to throw in some extra stuff (JVM builds, or maybe even 32-bit builds or something? You never know).
OK, so what can we do?
The most obvious one is to use compression in btrfs. The problem is that it is applied for each file individually, so we are not going to save anything across many builds. Also, it is only viable if you already have btrfs, so it looks like it is not the best option.
While it may sound like a great idea to compress all builds together, it does not work that well in practice. Well, it does, but keep reading.
The best compression I got is with 7z
. Each build is â4 MB (â28 MB uncompressed, therefore â7 times space saving!)
Compressing each build individually is also good for things like local bisect. That is, we can make these archives publicly available, and then write a script that will pull these archives for your local git bisect. How cool is that! That will be about 40 MB of stuff to download per git bisect, and you cannot really compress it any further anyway because you don't know ahead of time which files you would need.
This gives us about â120 GB per 10 years. Good enough, I like it.
Is there anything that performs better than 7z? Well, yes:
27M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar
9.0M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lz4
7.0M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.zip
6.7M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.gz
6.7M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lzfse
6.4M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.bz2
5.7M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7-L15.tar.zst
4.9M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7-L22.tar.zst
4.7M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lzham
4.5M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.brotli
4.3M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.lrz
4.1M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.7z
4.1M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tpxz
4.0M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7.tar.xz
3.8M c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7-extremely-slow.tar.lrz
xz
is much slower during compression and is tiny bit slower when decompressing, so the win is insignificant. lrz
with extreme options is much slower at everything, so forget it.
For the tests, I took only 7 builds:
c587b9d3e6f9c34819e2e8a9d63b8dc98a20a6d7
3878066a953195276ef99739f157d38793395d06
ef04e1e07f1de0d4eb2666985c7290f96c912be6
eea786e2a2febef0ab0bfabca956beae95ab81fd
76be77c9d6e697c26e92dc704109b7b8780845aa
d30806bd00dee201cd891550ac65f084f18e8285
9bfbab9186d710e0603b1eb86be1e5ba2e0c84d1
Now, there are some funny entries here. Obviously, .tar
is the size of all repos together, uncompressed. git-repo
is a git repo with every build committed one after another in the right order. Look, it performs better than some of the other things! Amazing! And even better if you compress it afterwards. Who would've thought, right?
187M all.tar
63M all.tar.lz4
49M all.zip
47M all.tar.gz
45M all.tar.bz2
42M git-repo.tar
35M all.tar.zstd # -19
28M all.tar.xz
28M all.7z
19M git-repo.7z
9.5M all.tar.zstd # -19 --long
8.3M all.tar.lrz
7.3M all.tar.zstd # --ultra -22 --long=31 --format=lzma # but it's much slower than lrz when compressing
However, lrz
clearly wins this time. And that's with default settings! Wow.
I think that there are ways to fiddle with options to get even better results. Suggestions are welcome!
However, at this point it looks like the best way is to use 7z
to compress each build individually.
Messing around with one huge archive is probably not worth the savings. It will make decompression time significantly slower, but we want decompression to be as fast as possible.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
đ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. đđđ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google â€ïž Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.