The cargo-port from galaxyproject

Python 2 only?

Given the following error:

$ galaxy-package-locator 
Traceback (most recent call last):
  File "/home/simon/anaconda3/bin/galaxy-package-locator", line 4, in <module>
    __import__('pkg_resources').run_script('cargo-port==1.0', 'galaxy-package-locator')
  File "/home/simon/anaconda3/lib/python3.5/site-packages/pkg_resources/__init__.py", line 738, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/simon/anaconda3/lib/python3.5/site-packages/pkg_resources/__init__.py", line 1506, in run_script
    exec(script_code, namespace, namespace)
  File "/home/simon/anaconda3/lib/python3.5/site-packages/cargo_port-1.0-py3.5.egg/EGG-INFO/scripts/galaxy-package-locator", line 3, in <module>
    __requires__ = 'cargo-port==1.0'
ImportError: No module named 'urllib2'

I suspect it's currently Python 2 only? Might be good to specify that.

More information from Travis CI if checksums fail

See discussion on #84 where TravisCI reports a failed checksum for an updated URL, but does not tell us anything about why. Possible failures:

Error 404 or similar (no file)
Wrong file, in which case would want to see the checksum and ideally file size, perhaps also output of file in case of things like an HTML or text error message?

Mirror CPC with universities/provide some redundancy

We need to (eventually) provide some redundancy for the CPC.

This will involve a couple things:

rsync (or other endpoint, some way to mirror things when the CPC updates)
various volunteers offering to host a complete copy of the CPC with their universities/etc
a new galaxy tag to support CPC downloads where we can update Galaxy on the backend to specify CPC hosts that can be used.

conda-build 3 recipes may have lists of sources

conda-build >=3 added the ability to use multiple sources when building a package by allowing a recipe's source section to be a list, optionally, see https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html#source-from-multiple-sources.
This is currently not handled by .conda/get_urls.py and .conda/to_cargoport.py.

TCP Missing a test suite

Training Data file

separate out training data
choose subset of columns
webpage support
gsl/other script support?

galaxy-package-locator fails (even on given example)

./bin/galaxy-package-locator --package_id samtools --package_version "1.2"

INFO:root:Searching for package: samtools in https://depot.galaxyproject.org/software/urls.tsv
Traceback (most recent call last):
  File "./bin/galaxy-package-locator", line 58, in <module>
    get()
  File "/home/berntm/.venv3/lib/python3.6/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/berntm/.venv3/lib/python3.6/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/berntm/.venv3/lib/python3.6/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/berntm/.venv3/lib/python3.6/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "./bin/galaxy-package-locator", line 36, in get
    for ld in yield_packages(handle):
  File "/home/berntm/projects/cargo-port/cargoport/utils.py", line 18, in yield_packages
    if line.startswith('#'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

Problem is that urllib opens with "rb", but if a file is given its "r".

I guess the best is to open always with "rb" and fix the startwith calls?

Expose build # / version number

Should:

link to latest build results on jenkins
allow users to tell if their mirror is out of date

Do not allow overwriting files

@natefoo could you add a sudoers entry for the user jenkins SSHs as into depot, for the following command:

sudo chattr -i /srv/.... (where ... is the path to cargo port's software directory).

How to tag files with multiple platform binaries?

Bad hashes in log on jenkins

https://jenkins.galaxyproject.org/job/cargo-port/lastBuild/console

E.g.

INFO:root:Mimetype of samtools/samtools_1.9_src_all.tar.bz2 is samtools/samtools_1.9_src_all.tar.bz2:	bzip2 compressed data , block size = 900k
ERROR:root:File has bad hash! Refusing to serve this to end users.

Fix CI tests

Instead of using curl, we should use urllib2 like we do in cargo port.

FCGI Incorrect

FCGI has a bad hash. It looks like upstream replaced the package, and we didn't have a good copy to start with. What should we do?

Bad hash, 6125df9285fc22f1add8062ba06248a41b291659997948c670ead21aca943a73 != 08bf8afe1985b17d2375e8dc98ac86cab0daf017ccd5281157abdbb7e19273fd in FCGI/FCGI_0.67_src_all.tar.gz

How to contribute

Hi, how is it be possible to upload new packages?

A Galactic Name

Proposed:

Supernova
spaceport
The Cargo Bay
SpaceCargo Bay
Cargo Dock

Rejected:

Cargo Hub (docker hub)
The Cargo Bay (TBP)
Supernova (??)

Considered:

Space Port
Cargo Dock
Cargo Port

Since it's a mirror we could go more syfy and be inspired by Dyson Spheres or maybe famous telescopes.

Feel free to submit ideas and discuss them! Something will be selected before the move into the galaxyproject namespace in approximately a week (I'd like it to generally be done around the same time as the IUC PR)

ccing @davebx @martenson @blankenberg

recommendation for unique package identifier

We need a recommendation for a package naming schema.
name _ major-version _ minor-version?

Rename packages for transparency in URL use.

at https://depot.galaxyproject.org/software/ if you download some package e.g. https://depot.galaxyproject.org/software/30e84ac4b31d40b66293f7b1221331c5a50561a39d580d85004d9c1fff991751

you will get a file named 30e84ac4b31d40b66293f7b1221331c5a50561a39d580d85004d9c1fff991751 that does not have any extension

This is quite confusing.

Upstream JS resource has bad SRI hash, datatables not loading on cargo-port

getting js errors when accessing https://depot.galaxyproject.org/software/ and the cached URL list is empty

Rsync URL

I know you'd like to do stuff with CVMFS and proper removal of SPOF, but if that isn't a short term solution, is there an rsync URL available? I'd like to ensure we at least have a complete clone of this somewhere (with the knowledge that we'll do DNS round robin and etc. later)

GSL.py needs some love

Currently gsl.py downloads a copy of urls.tsv and then sorts through packages for the one desired.

It could be improved a fair bit:

cache copy of urls.tsv, update that only if it has changed (nginx provides an ETag header for that file :D), we can use that to check if we need to fetch the whole file. (Think apt-get update)
the install script needs to expose more flags
- package name
- version
- architecture
- attempt to fetch upstream or just use depot

Safety mitigations following recent package loss incident

At least one extant backup #95
Full CVMFS / dns RR solution
do not allow editing files #96

SHA256SUMS are BIN files

If I open, for example, https://depot.galaxyproject.org/software/abyss/ with Firefox and click on the SHA256SUMS file, the browser wants to download it instead of showing the contents because it is recognized as a BIN file.

zlibbioc 1.14.0 checksum mismatch

At the website (https://depot.galaxyproject.org/software/) cargo port says:

<action type="download_by_url" sha256sum="213cec750b14254df25e160463e28f3857ba703df1f3f43f7082dece8cfbf08b">
    https://depot.galaxyproject.org/software/zlibbioc/zlibbioc_1.14.0_src_all.tar.gz
</action>

This checksum should be 0811b4aecdb66a2ee33e806a40f54e0fdde2f4223a3981cccc16df9b7d3567d4. It looks like the system confuses the following two files:

0811b4aecdb66a2ee33e806a40f54e0fdde2f4223a3981cccc16df9b7d3567d4  zlibbioc_1.14.0_src_all.tar.gz
213cec750b14254df25e160463e28f3857ba703df1f3f43f7082dece8cfbf08b  zlibbioc_1.14.0.tar.gz

Licensing and traffic redirection issues

Not sure if any of these two is a big issue but they should be considered.

some binaries/sources have restrictive licenses that might prevent redistribution from depot
we will affect metrics of the tool creators/hosts by redirecting their traffic to us

Webhooks

@bgruening shouldn't have to tell people "no later than one hour." This should happen automatically. Jenkins should tell users their package(s) are available.

cc @martenson

Store files in CVMFS

Heyyyyyyy @natefoo

CI reports on new packages

List tarball contents/filetypes/sizes/etc.

cc @bgruening @martenson

Sort urls.tsv

Sadly sort urls.tsv and even sort -n urls.tsv don't handle the version column nicely, but would be a simple way to define a consistent order for the file which would be easy to enforce on TravisCI, and apply when making manual additions.

Instead of generating individual hash files, provide a single cpc.sha256sum file

better UX
simplifies web interface

Add other depot packages to the download cache

e.g. should the urls from https://depot.galaxyproject.org/software/ be available in the file thus making it the universal resource for package location?

Archiving wheels

I'd like to archive h5py for galaxyproject/galaxy#4519, but I don't think it'd be possible with the current urls.tsv format because linux builds have multiple versions for each CPython ABI (basically the default Unicode width, cp27m vs cp27mu).

I could also put them on wheels.galaxyproject.org, but cargo port seems like the better long term solution, especially since we did not generate the wheels.

Any ideas? I could add it to the version field, I guess? e.g. 2.7.1-cp27mu?

galaxyproject / cargo-port Goto Github PK

cargo-port's People

Contributors

Stargazers

Watchers

Forkers

cargo-port's Issues

Recommend Projects

Recommend Topics

Recommend Org