The new jsDelivr backend automatically keeps all projects updated by pulling files directly from npm and github.
jsdelivr / libgrabber Goto Github PK
View Code? Open in Web Editor NEWDEPRECATED Keeps projects hosted in jsDelivr updated
DEPRECATED Keeps projects hosted in jsDelivr updated
The new jsDelivr backend automatically keeps all projects updated by pulling files directly from npm and github.
I need libgrabber to grab the following url for handsontable to be updated by jsdelivr
https://api.github.com/repos/handsontable/jquery-handsontable/tags?per_page=200
There are a lot of tags and they are not being returned in an order that the default pagination will list the latest ones.
npm WARN deprecated [email protected]: this package has been reintegrated into npm and is now out of date with respect to npm
npm WARN deprecated [email protected]: graceful-fs version 3 and before will fail on newer node releases. Please update to graceful-fs@^4.0.0 as soon as possible.
npm WARN deprecated [email protected]: Jade has been renamed to pug, please install the latest version of pug instead of jade
npm WARN deprecated [email protected]: graceful-fs version 3 and before will fail on newer node releases. Please update to graceful-fs@^4.0.0 as soon as possible.
npm WARN deprecated [email protected]: lodash@<3.0.0 is no longer maintained. Upgrade to lodash@^4.0.0.
npm WARN deprecated [email protected]: graceful-fs version 3 and before will fail on newer node releases. Please update to graceful-fs@^4.0.0 as soon as possible.
npm WARN engine [email protected]: wanted: {"node":"0.8.x"} (current: {"node":"4.4.3","npm":"2.15.1"})
npm WARN engine [email protected]: wanted: {"node":"0.8.x"} (current: {"node":"4.4.3","npm":"2.15.1"})
npm WARN engine [email protected]: wanted: {"node":"0.8.x"} (current: {"node":"4.4.3","npm":"2.15.1"})
npm WARN engine [email protected]: wanted: {"node":"0.8.x"} (current: {"node":"4.4.3","npm":"2.15.1"})
npm WARN engine [email protected]: wanted: {"node":"0.8.x"} (current: {"node":"4.4.3","npm":"2.15.1"})
Several projects maintain multiple major version (notably jQuery). It'd be useful to support syncing latest tags on different major versions
We accidentally added an update.json to a project that doesnt have any tagged releases. So libgrabber crashed. Can we catch this kind of errors and simply skip these projects?
I noticed that project.js looks at SemVer to flag updates, which is great.
Should it be in scope to have a new update be pushed without a SemVer increase? Sometimes a dev might forget a minor change, then add it & push out an update with out feeling they need to increase the SemVer.
If its possible it would help a lot to add a parameter to update.json.
Something like "auto-merge": true
.
It can be enabled per project and if the bot sees it then it will automatically merge the PR for that project without waiting for a mod.
So far the bot already has 3 branches + master.
On every new run can we delete old branches and leave only master?
I'm interested in hearing your opinions about how libgrabber should approach with pushing project updates to git. My idea is that libgrabber doesn't work directly with jsdelivr/jsdelivr
repo, but uses its own fork (with a separate github account). For example, let say that librabber found a new version in one of the projects, then it does following things:
<project name>/<version>
<project name>/<version>
into jsdelivr/jsdelivr
Every time libgrabber get scheduled to run, it merges changes from upstream (jsdelivr/jsdelivr
). On the next execution it will not update same version again because the pull request is accepted by @jimaek and new version is found in the file system.
If the pull request is still not accepted, then libgrabber checks if there is a remote branch (<project name>/<version>
) in its own fork and skips updating project again. This works rather well in cases when there is an error in update.json or a bug in libgrabber. You could just fix update.json and delete remote branch directly from github, libgrabber will then try to update the project again.
This is implemented as of now, with the exception of doing pull requests. There are couple of branches created in test jsdelivr fork https://github.com/aleksandarabot/jsdelivr/branches.
Please tell me your suggestions and improvements to this workflow.
If a project doesn't have any local versions then libgrabber should simply download the latest available.
Example:
files/jquery.cookie/
info.ini
update.json
This is going to be awesome as it doesnt require the author to fork jsDelivr and add an initial version. He can simply use Web GUI to add update.json and libgrabber will take care of the rest.
https://github.com/zmmbreeze/lining.js/releases/tag/0.3.1 is up for a day, but https://github.com/zmmbreeze/lining.js/blob/master/build/lining.min.js etc still have v0.3.0
. I assume you either didn't bump package.json
until after you ran Grunt, or forgot to upload the new files?
Likely a v2 thing, but could work something like this:
Update.json
README.*
Unfortunately, badges can't be used anymore for notifications.
Fetch files off release
When libgrabber checks if project is updated, it reads metadata from update.json
. Metadata contains information like name of package, type of package manager (npm or bower), repository url (if project is not using package manager) and a list of files to be included or excluded.
For example, for a project that uses npm or bower:
{
"packageManager": "npm" // "bower"
"name": "lodash",
"files": {
"basePath": "dist/",
"include": ["*.js"],
"exclude": ["lodash.underscore.js", "lodash.underscore.min.js"]
}
}
If the project is not using package manager, then the repository
value identifies project location. For example:
{
"repository": "https://github.com/lodash/lodash"
"files": {
"basePath": "dist/",
"include": ["*.js"],
"exclude": ["lodash.underscore.js", "lodash.underscore.min.js"]
}
}
packageManager
- type of package manager (npm or bower)
name
- package name
repository
- required if packageManager
or name
are not defined, refers to the project's git repository
files.basePath
(optional) - base path to use when looking for resources
files.include
(optional, multiple) - files or paths to include
files.exclude
(optional, multiple) - files or paths to exclude
Includes/excludes might be specified using glob (see https://github.com/isaacs/node-glob).
Many smaller projects, particularly those of CSS and JavaScript such as hosted on jsDelivr, utilize a major.minor structure. Despite not being SemVer compliant per-se, libgrabber should probably support them.
The way I would do this is to detect the versions with only one dot separating two integers early on, and appending a second minor version to fix the detection of new versions, but use the original for the folder names in the projects.
1.2 => 1.2.0
1.4.3 => 1.4.3
Lately the bot started closing update PRs immediately after opening them and deleting the branch. This had never happened before. Any ideas?
Lots of big projects want instant updates on our CDN.
Any ideas how to use Github's webhooks functionality to ping libgrabber every-time there is a new release and quickly update that project?
dist/*.js
- copies javascript files from dist dir. Directory structure will not be retained (e.g.dist
dir will be stripped when copied). To retain directory structure prepend glob with./
e.g../dist/*.js
What happens if all files for consumption are inside dist
, but they have a dist/vendors
folder, & they want only the /vendors
folder to be a directory in jsDelivr?
Eg: for "include": [ "thing.css", "thing..min.css", "thing.js", "thing.min.js"]
, if jquery.thing.js
's SemVer is changed, then all files are uploaded to a new version on jsDelivr, even if the other files' versions aren't bumped? How about if a non-mainfile version is bumped like a CSS?
Could be a race condition if a PR is accepted before the min files are updated.
Besides getting project updates from github, libgrabber supports popular package managers npm and bower.
I assume now that this means NPM or Bower may be used only to flag new releases, instead of Releases?
And the files are still uploaded directly from GitHub?
Is there any way to make the bot rename some files before pushing them?
This way we can make sure that all projects will follow our structure.
Discussion jsdelivr/jsdelivr#915
name
(required) - refers to package name on npm or Bower, or repo name when GitHub is used
Seems that name
really reflects jsDelvr's folder name if you look at this example? If true, then I'll update.
The bot should have an option to disable publishing of alpha-, beta- or rc-* releases
Hi there,
We've recently pushed the latest version of our javascript client: v2.6.6 and we'd like to make it available to our users through jsdelivr to always provide them our latest version, so they won't have to worry about minor updates.
We just discovered that aliases point to different versions, and not to the last one.
Latest version: 2.6.6 (added 13 days ago)
http://cdn.jsdelivr.net/algoliasearch/latest/algoliasearch.min.js -> 2.6.2
http://cdn.jsdelivr.net/algoliasearch/2/algoliasearch.min.js -> 2.6.5
Sounds like a caching issue. Could you please have a look into it.
Otherwise, can you tell me how often does the libgrabber bot crawle github repos?
Thanks,
I found a project that seemed to fail to upload the new files after bumping a new release. 2 clues:
I'd think it would be nice to at least warn of old files, so our admins won't push a non-update update when they're multitasking.
For example chartist changed their structure and libgrabber stopped updating it.
The only way to see the problem is to read the logs
info: Updating chartist.js to v0.4.3
info: Copied 0 files name=chartist.js, version=v0.4.3, files=[]
warn: Something might be wrong with update.json name=chartist.js, version=v0.4.3
an other example
error: Error in reading metadata file /home/jsdelivr/files/underscore.string/update.json
warn: Failed to read metadata metadataPath=/home/jsdelivr/files/underscore.string/update.json,
error: Failed to update project /home/jsdelivr/files/underscore.string/update.json
error: Failed to update project
There needs to be a way to automatically notify the admin via email about a library that fails to update
Not sure what the file cap should be?
We need a detailed readme for this project describing how to install and use the tool for people that want to contribute code and for the authors how to keep their projects auto-updated.
Some projects may not push releases on the "master" branch but rather do the dev work on the master branch. It would be great if repos with releases on other branches are supported.
I'm trying to understand; so people have to install libgrabber on their local machine?
For some reason I thought it was a jsDelivr bot...
As discussed on Gitter, I think it'd be useful to add support for ftp
and file index
s. For example, see angular's index.
Could make managing pull requests like this jsdelivr/jsdelivr#5335 significantly easier via auto-update.
/cc @jimaek @UnbounDev
For example this fails
{
"packageManager": "github",
"name": "matreshka",
"repo": "finom/matreshka",
"files": {
"include": ["matreshka.min.js", "matreshka.js", "matreshka.min.map"]
}
}
because finom/matreshka
is now matreshkajs/matreshka
The bot should be able to follow the redirects and use the new repo names.
https://github.com/wavded/humane-js
How do we include the themes folder?
{
"packageManager": "github",
"name": "humane.js",
"repo": "wavded/humane-js",
"files": {
"basePath": "/",
"include": ["humane.min.js", "humane.js", "themes"]
}
}
Doesnt work.
Updating using metadata/home/jsdelivr/files/humane.js/update.json
Read metadata
{ metadata:
{ packageManager: 'github',
name: 'humane.js',
repo: 'wavded/humane-js',
files:
{ basePath: '/',
include:
[ 'humane.min.js',
'humane.js',
'themes' ] },
path: '/home/jsdelivr/files/humane.js' },
versions:
{ localVersions: [ '3.0.6', '3.1.0' ],
remoteVersions:
[ '1.1.0',
'1.2.0',
'1.2.1',
'1.3.0',
'1.4.0',
'2.0.0',
'2.1.0',
'2.1.1',
'2.2.0',
'2.2.5',
'2.2.6',
'2.2.7',
'2.2.8',
'2.5.0',
'2.6.0',
'2.7.0',
'2.7.1',
'2.7.2',
'2.8.0',
'2.8.1',
'3.0.0',
'3.0.1',
'3.0.2',
'3.0.3',
'3.0.4',
'3.0.5',
'3.0.6',
'3.1.0',
'3.2.0' ],
branchVersions: [] } }
Updating humane.js to 3.2.0
Copied 3 files
{ name: 'humane.js',
version: '3.2.0',
files:
[ '/home/jsdelivr/files/humane.js/3.2.0/humane.min.js',
'/home/jsdelivr/files/humane.js/3.2.0/humane.js',
'/home/jsdelivr/files/humane.js/3.2.0/themes' ] }
And themes/*
would simply add the contents of the folder.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.