sckott/cchecksapi

CRAN checks API (DEFUNCT)

Hi, I'm Scott 👋

I'm a software engineer at Fred Hutch in the Fred Hutch Data Science Lab

• LinkedIn
• website
• blog
• mastodon - @[email protected]
• bluesky - sckott
• 🐝   Fun fact: ~90% of bee species are solitary (not social)

skills

 

cchecksapi's Issues

memtests failing?

hi @sckott can we query your db to get a list of all packages for which at least one of the memtests https://www.stats.ox.ac.uk/pub/bdr/memtests/README.txt fails?

Add summary about checks across platforms to /pkgs

To make it quick to see if there's >0 or 0 errors/warnings/notes

check_url is not being collected for some pkgs

curl https://cranchecks.info/pkgs/urltools | jq '.data.checks[].check_url'

"https://www.R-project.org/nosvn/R.check/r-release-osx-x86_64/urltools-00check.html"
"https://www.R-project.org/nosvn/R.check/r-oldrel-osx-x86_64/urltools-00check.html"
null
null
null
null
null
null
null
null
null
null

Badges?

Maybe a single combined badge that shows the "worst" result?

FWIW here is some example badge generation code. Different language, but maybe still useful:
https://github.com/metacran/metacranweb/blob/8e930ab92c2f73c0cef4c3f64cf9ae208aa0bb9f/routes/badges.js

move to Ruby 3.1

error correctly on limit/offset values that are not integers

Could date updated come earlier

Looking at e.g. "https://cranchecks.info/pkgs/rhub/history" I was wondering whether within each slot, "date_updated" could come before "summary"?

routes for notifications

brainstorming.

/notifications would be landing page and user interface page for managing notifications

probably all below would be auth protected

• GET /notifications/rules
• POST /notifications/rules w/ body for a new rule
• GET/DELETE/PUT /notifications/rules/{rule id}

---

• POST /notifications/rules

[
  {"package": "foobar", "status": "error", "time": 3, "platforms": 2},
  {"package": "helloworld", "status": "warn", "platforms": "osx"},
  {"package": "bloop", "regex": "install failure"}
]

run rake tasks (updates) in a different container?

right now rake tasks are run inside the api container. guessing this is causing problems - possibly run rake tasks in a linked container thats somehow cleaned up memory/cpu wise after tasks are done

copy-pasting from Fishbase?

There are two occurrences of copy-pasting from Fishbase API docs

https://github.com/ropenscilabs/cchecksapi/blob/master/docs/api_docs.md#response-bodies

https://github.com/ropenscilabs/cchecksapi/blob/master/docs/api_docs.md#heartbeat

blah, maintainers scraper is failing on server but works locally

...

change heartbeat route to automagically get routes

same as sckott/bienapi#25, see solution https://github.com/ropenscilabs/bienapi/blob/ca093751231decb343d75e7fdca1b39becbb1a86/api.rb#L167

lowercase all keys

API for maintainers

I.e. these kind of pages:
https://cran.r-project.org/web/checks/check_results_csardi.gabor_at_gmail.com.html

Would be nice. :)

history: collect all history ?

In addition to new history routes, maybe push all history data to disk and provide a dump? so when history data is 30 days old purge from mongo and compress onto disk - so a file for each date, like

• cran-checks-2018-10-01.json
• cran-checks-2018-10-02.json
• cran-checks-2018-10-03.json
• and so on

then a route for requesting these files

/pkghistory/:date: - e.g., /pkghistory/2018-10-01

Allow regular email slugs in addition to cran type slugs in /maintainers route

so [email protected] instead of csardi.gabor_at_gmail.com

maybe grab copy of cran and scrape from it locally

instead of scraping via web requests - maybe too much complexity

https://cran.r-project.org/mirror-howto.html

email frequency?

probably only send the same email about the same rule once per week? or once every 3 days or so?

limit param: make limit=0 return no results, but do populate records found field

status=NOTE URLs aren't being scraped correctly

fill /maintainers table summaries with all keys

so every hash has all the same keys

Consider Redis caching for some routes

in particular for

    "/pkgs (GET)",
    "/pkgs/:pkg_name:/history (GET)",
    "/pkghistory (GET)",
    "/maintainers (GET)",
    "/maintainers/:email: (GET)",

as these can have large payloads, and could be cached to minimize response time

Ideally: check if the cache should be deleted if there's been a recent cron job run resulting in new data for whatever the query is.

Notifications

Want to allow people to opt in to notifications of various kinds to find out when theres notes/warnings/errors on their pkgs on cran checks.

Options:

• email
• twitter?
• slack?
• any others?

Probably should set up super minimal website for it. Pick which type of notification, which pkgs, on what type (warn/note/error).

`/history` route

looking into it. work on history branch.

URLs won't be useful as they'll just point to the latest checks, so need to scrape the checks text for each URL

check url's not being fetched correctly

switch from Unicorn to Puma server

https://github.com/puma/puma

Scraping maintainers frozen at 3 April 2019?

Hello,

I've noticed something a bit odd, it seems that maintainer scraping has been frozen at 2019-04-03.

$ curl https://cranchecks.info/maintainers/maelle.salmon_at_yahoo.se | jq '.data.date_updated'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1779  100  1779    0     0   2691      0 --:--:-- --:--:-- --:--:--  2691
"2019-04-03T12:01:43.725Z"

$ curl https://cranchecks.info/maintainers/zkamvar_at_gmail.com | jq '.data.date_updated'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1242  100  1242    0     0   1040      0  0:00:01  0:00:01 --:--:--  1040
"2019-04-03T12:01:43.744Z"

$ curl https://cranchecks.info/maintainers/hadley_at_rstudio.com | jq '.data.date_updated'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9913  100  9913    0     0  13981      0 --:--:-- --:--:-- --:--:-- 13981
"2019-04-03T12:01:43.806Z"

$ curl https://cranchecks.info/maintainers/edd_at_debian.org | jq '.data.date_updated'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 14573  100 14573    0     0  14301      0  0:00:01  0:00:01 --:--:-- 14301
"2019-04-03T12:01:43.793Z"

Is there something I'm doing wrong on my side?

Badges state

Is it binary is it OK vs not OK?

emails should have html and plain text versions

see emails from r-hub as well as from travis-ci for how that's done. travis e.g.,

Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
# vs. 
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: quoted-printable

in sendgrid-ruby gem, it looks like we can do something like

x = Content.new(type: 'text/html', value: '<html>some string</html>')
x = SendGrid::Content.new(type: 'text/plain', value: "another string")

maybe scrape results for individual checks?

Use case in weathercan: apparently sometimes the check errors/warnings are not included in the check_details bit at the bottom of the check results landing pages

in https://cranchecks.info/pkgs/weathercan/history?limit=30 there's errors, but only notes about non-ascii characters are shown in the check_details instead of the error details

This would be a lot of extra data if we did for every check for every pkg. Could do for only errors for each pkg?

/pkgs/:pkg_name:/history -> sort by date descending

fix how we ge pkg names

right now we're topping out at 12K

add base url for checks page to output for easy access

how best to include output if error/warn

Would be nice to give users of the API some of the actual output if there are check failures/warnings. However, not sure how to include e.g, this

in JSON 🤔

convert formats

numbers are numbers, etc.

should archived pkgs be removed?

e.g., i just archived this pkg https://cranchecks.info/pkgs/crevents and the data is still there, i guess each day now has the last day of data that existed?

often though pkgs can be archived by cran, then submitted again, etc. - so we wouldn't want to lose history for that pkg.

parallel gem in_processes - try tweaking to see if helps with RAM issues

memory limit

the api container seems to take up a lot of memory, perhaps set limit to lower than what it is. it's at 1.95 GB memory right now, maybe lower to 1.2 or so

get sidekiq working for scraping once a day

move all history data into sql DB?

Maybe it will help with memory issues - and can easily fit the data into tabular structure:

pkg	summary	checks	check_details	date_updated
webmockr	summary_json	checks_json	check_details_json	2018-11-16 21:04:00 UTC

history: remove oid field in results

change date_created to date_updated

Change "warning" in /pkgs summary hash to "warn"

to make consistent with maitnainers output, and "warn" is what CRAN uses as well

change port configuration on mongodb docker compose file before pulling from master

see server

CRANCHECK badge not working?

Got one question about CRAN CHECKS badges managed by rOpenSci. I wonder why I got an unknown badge while official CRAN production is all OK?

URL I use for badge is
https://cranchecks.info/badges/summary/wyz.code.offensiveProgramming
URL I use for production check is
https://cran.r-project.org/web/checks/check_results_wyz.code.offensiveProgramming.html

Any reason for such behavior?
I probably missed a point. Hints are welcome

scraping maintainers: use gabor's api

see r-hub/crandb#45 (comment)

right now in https://github.com/ropensci/cchecksapi/blob/rework-scraping/scrape_maints.sh scraping from the CRAN maintainers overview page, but that's slow.

make sure gabor's /maintainernames route has up to date list of maintainers

use trailing slash helper to avoid those stupid forbidden errors

Add "any" entry to /maintainers route to summarise results

Ability to ignore some NOTEs in badges

via #28

continuing conversation here

new ganda version

https://github.com/tednaleid/ganda/releases/tag/v0.1.7

beware, looks like need to ask for parallelism, make sure to test taht before moving over to new version