Sometimes returns 403 Forbidden from GH

[plicease]

Sometimes returns 403 Forbidden from GH

[plicease]

I wonder if this is a rate limiting by GH?

[plicease]

So yeah. This is a problem which may render this plugin pretty useless:

{
  "message": "API rate limit exceeded for x.x.x.x. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)",
  "documentation_url": "https://developer.github.com/v3/#rate-limiting"
}

[kiwiroy]

Only a valid personal access token (PAT) with public_repo no scopes is required to increase that limit to 5000/hour.

# export a valid token from https://github.com/settings/tokens
export GITHUB_PAT=0955...1223
curl -i -H "Authorization: token $GITHUB_PAT" https://api.github.com/repos/Perl5-Alien/dontpanic/releases | head -n10
HTTP/1.1 200 OK
Date: Tue, 30 Apr 2019 11:22:24 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 7735
Server: GitHub.com
Status: 200 OK
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4999
X-RateLimit-Reset: 1556626901
Cache-Control: private, max-age=60, s-maxage=60

It would be possible for something like:

plugin 'Download::GitHub' => (
    github_user => 'Perl5-Alien',
    github_repo => 'dontpanic',
    token => $ENV{GITHUB_PAT}
);

However, Alien users as well as authors would need to know about export GITHUB_PAT=... if they are installing more than 60 Aliens using this plugin per hour - this would include cpantesters.

edit See e0020f4 for an option that doesn't require having headers set in Plugin::Fetch::*.

[plicease]

@kiwiroy I think it is reasonable for the plugin to default to a token specified in the environment without having to pass it in from the alienfile. In fact it I don't think it ever makes sense to specify the token in the alienfile? This would definitely be an improvement, although it still has the problem that most testers will not have the environment variable set.

[plicease]

One security challenge will be having a security credential in the URL because the URLs are often displayed in diagnostic messages. Either using the header method, or a way to mask the URL. Neither of these are built into AB core yet.

[eserte]

Another possibility would be the usage of a cache directory for the downloaded distfiles (possibly size-limited using LRU mechanisms or so). This could help on smokers or other build systems sharing the same home directory.

[plicease]

@eserte I started working on a plugin to cache requests here:
https://metacpan.org/pod/Alien::Build::Plugin::Fetch::Cache
as the documentation states it is pretty dumb and probably not ready for prime time, but with some extra shine could probably be useful for the use cases you suggest.

[plicease]

With #13 and #14, I think this is solved as well as can be currently. We can revisit and open additional if future techniques can be used to improve further.

@eserte This will work with AB 2.39+ and ABP::Download::GitHub 0.07, but note the caveats.