I'm a software engineer at Fred Hutch in the Fred Hutch Data Science Lab
- website
- blog
- mastodon - @[email protected]
- bluesky - sckott
- ๐ ย Fun fact: ~90% of bee species are solitary (not social)
CRAN checks API (DEFUNCT)
Home Page: https://github.com/r-hub/cchecksbadges
License: MIT License
I'm a software engineer at Fred Hutch in the Fred Hutch Data Science Lab
hi @sckott can we query your db to get a list of all packages for which at least one of the memtests https://www.stats.ox.ac.uk/pub/bdr/memtests/README.txt fails?
To make it quick to see if there's >0 or 0 errors/warnings/notes
curl https://cranchecks.info/pkgs/urltools | jq '.data.checks[].check_url'
"https://www.R-project.org/nosvn/R.check/r-release-osx-x86_64/urltools-00check.html"
"https://www.R-project.org/nosvn/R.check/r-oldrel-osx-x86_64/urltools-00check.html"
null
null
null
null
null
null
null
null
null
null
Maybe a single combined badge that shows the "worst" result?
FWIW here is some example badge generation code. Different language, but maybe still useful:
https://github.com/metacran/metacranweb/blob/8e930ab92c2f73c0cef4c3f64cf9ae208aa0bb9f/routes/badges.js
Looking at e.g. "https://cranchecks.info/pkgs/rhub/history" I was wondering whether within each slot, "date_updated" could come before "summary"?
brainstorming.
/notifications
would be landing page and user interface page for managing notifications
probably all below would be auth protected
/notifications/rules
/notifications/rules
w/ body for a new rule/notifications/rules/{rule id}
/notifications/rules
[
{"package": "foobar", "status": "error", "time": 3, "platforms": 2},
{"package": "helloworld", "status": "warn", "platforms": "osx"},
{"package": "bloop", "regex": "install failure"}
]
right now rake tasks are run inside the api container. guessing this is causing problems - possibly run rake tasks in a linked container thats somehow cleaned up memory/cpu wise after tasks are done
There are two occurrences of copy-pasting from Fishbase API docs
https://github.com/ropenscilabs/cchecksapi/blob/master/docs/api_docs.md#response-bodies
https://github.com/ropenscilabs/cchecksapi/blob/master/docs/api_docs.md#heartbeat
...
I.e. these kind of pages:
https://cran.r-project.org/web/checks/check_results_csardi.gabor_at_gmail.com.html
Would be nice. :)
In addition to new history routes, maybe push all history data to disk and provide a dump? so when history data is 30 days old purge from mongo and compress onto disk - so a file for each date, like
cran-checks-2018-10-01.json
cran-checks-2018-10-02.json
cran-checks-2018-10-03.json
then a route for requesting these files
/pkghistory/:date:
- e.g., /pkghistory/2018-10-01
so [email protected]
instead of csardi.gabor_at_gmail.com
instead of scraping via web requests - maybe too much complexity
probably only send the same email about the same rule once per week? or once every 3 days or so?
so every hash has all the same keys
in particular for
"/pkgs (GET)",
"/pkgs/:pkg_name:/history (GET)",
"/pkghistory (GET)",
"/maintainers (GET)",
"/maintainers/:email: (GET)",
as these can have large payloads, and could be cached to minimize response time
Ideally: check if the cache should be deleted if there's been a recent cron job run resulting in new data for whatever the query is.
Want to allow people to opt in to notifications of various kinds to find out when theres notes/warnings/errors on their pkgs on cran checks.
Options:
Probably should set up super minimal website for it. Pick which type of notification, which pkgs, on what type (warn/note/error).
looking into it. work on history
branch.
URLs won't be useful as they'll just point to the latest checks, so need to scrape the checks text for each URL
Hello,
I've noticed something a bit odd, it seems that maintainer scraping has been frozen at 2019-04-03.
$ curl https://cranchecks.info/maintainers/maelle.salmon_at_yahoo.se | jq '.data.date_updated'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1779 100 1779 0 0 2691 0 --:--:-- --:--:-- --:--:-- 2691
"2019-04-03T12:01:43.725Z"
$ curl https://cranchecks.info/maintainers/zkamvar_at_gmail.com | jq '.data.date_updated'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1242 100 1242 0 0 1040 0 0:00:01 0:00:01 --:--:-- 1040
"2019-04-03T12:01:43.744Z"
$ curl https://cranchecks.info/maintainers/hadley_at_rstudio.com | jq '.data.date_updated'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9913 100 9913 0 0 13981 0 --:--:-- --:--:-- --:--:-- 13981
"2019-04-03T12:01:43.806Z"
$ curl https://cranchecks.info/maintainers/edd_at_debian.org | jq '.data.date_updated'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 14573 100 14573 0 0 14301 0 0:00:01 0:00:01 --:--:-- 14301
"2019-04-03T12:01:43.793Z"
Is there something I'm doing wrong on my side?
Is it binary is it OK vs not OK?
see emails from r-hub as well as from travis-ci for how that's done. travis e.g.,
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
# vs.
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable
in sendgrid-ruby gem, it looks like we can do something like
x = Content.new(type: 'text/html', value: '<html>some string</html>')
x = SendGrid::Content.new(type: 'text/plain', value: "another string")
Use case in weathercan: apparently sometimes the check errors/warnings are not included in the check_details
bit at the bottom of the check results landing pages
in https://cranchecks.info/pkgs/weathercan/history?limit=30 there's errors, but only notes about non-ascii characters are shown in the check_details instead of the error details
This would be a lot of extra data if we did for every check for every pkg. Could do for only errors for each pkg?
right now we're topping out at 12K
numbers are numbers, etc.
e.g., i just archived this pkg https://cranchecks.info/pkgs/crevents and the data is still there, i guess each day now has the last day of data that existed?
often though pkgs can be archived by cran, then submitted again, etc. - so we wouldn't want to lose history for that pkg.
the api container seems to take up a lot of memory, perhaps set limit to lower than what it is. it's at 1.95 GB memory right now, maybe lower to 1.2 or so
Maybe it will help with memory issues - and can easily fit the data into tabular structure:
pkg | summary | checks | check_details | date_updated |
---|---|---|---|---|
webmockr | summary_json | checks_json | check_details_json | 2018-11-16 21:04:00 UTC |
to make consistent with maitnainers output, and "warn" is what CRAN uses as well
see server
Got one question about CRAN CHECKS badges managed by rOpenSci. I wonder why I got an unknown badge while official CRAN production is all OK?
URL I use for badge is
https://cranchecks.info/badges/summary/wyz.code.offensiveProgramming
URL I use for production check is
https://cran.r-project.org/web/checks/check_results_wyz.code.offensiveProgramming.html
Any reason for such behavior?
I probably missed a point. Hints are welcome
right now in https://github.com/ropensci/cchecksapi/blob/rework-scraping/scrape_maints.sh scraping from the CRAN maintainers overview page, but that's slow.
make sure gabor's /maintainernames route has up to date list of maintainers
via #28
continuing conversation here
https://github.com/tednaleid/ganda/releases/tag/v0.1.7
beware, looks like need to ask for parallelism, make sure to test taht before moving over to new version
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.