Giter Site home page Giter Site logo

nebulizer's People

Contributors

hugovk avatar pjbriggs avatar

Stargazers

avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

hugovk

nebulizer's Issues

Easier management of API keys within nebulizer

It would be nice to be able to create and update Galaxy instance references and API keys in ~/.nebulizer more automatically, for example could I do:

nebulizer add main http://usegalaxy.org/ [email protected]

and then have it prompt me for my password, retrieve my API key and then add this to the .nebulizer file?

'update_tool': uncaught JSONDecodeError exception

Attempting to update iuc/trinity_analyze_diff_expr using Nebulizer 0.6.0 gets an uncaught JSONDecodeError exception which crashes Nebulizer (the operation still completes in Galaxy).

The problem occurs when the connection with the initial tool install attempt is broken, presumably all that is required is an extension to the exception handling in the install_tool function:

...
trinity_analyze_diff_expr: requesting installation
WARNING:nebulizer.tools:Got error from Galaxy API on attempted install (ignored)
WARNING:nebulizer.tools:Status code: 504
Traceback (most recent call last):
  File "/home/pjb/virtual-envs/nebulizer/bin/nebulizer", line 8, in <module>
    sys.exit(nebulizer())
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.    6/site-packages/nebulizer/cli.py", line 887, in update_tool
    (install_resolver_dependencies== 'yes')))
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/nebulizer/tools.py", line 1322, in update_tool
    no_wait=no_wait)
  File "/home/pjb/virtual-envs/nebulizer/lib/python3.6/site-packages/nebulizer/tools.py", line 1189, in install_tool
    json.loads(connection_error.body)["err_msg"])
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Merge tool/user/library utility functionality into subcommands of 'nebulizer'

Currently there are four utilities: nebulizer, manage_users, manage_tools and manage_libraries.

It would be more consistent with other packages such as planemo to provide a single utility nebulizer which offered all the functionality from the other utilities as subcommands - for example

manage_users list -> nebulizer list_users
manage_users create -> nebulizer create_user

and so on.

Fetching API key from Galaxy instance doesn't work when user is not admin

The Bioblend galaxy.users.UserClient(gi).create_user_apikey(user_id) function can only be invoked by an admin user, so it won't work for fetching a new API key on Galaxy instances where the user is not an admin.

As an alternative it looks like the galaxyclient.GalaxyClient class (which GalaxyInstance subclasses) implements a key property which returns the API key for the current user.

Gracefully handle server/proxy timeouts when installing tool repository

For tool repositories with large numbers of dependencies, it appears that the proxy server can timeout before Galaxy has finished processing the dependency tree and returned a response.

In these cases Nebulizer receives a 502 error from Bioblend, which is dumped to stdout - however it would be nicer if this could be gracefully handled with a suitable message instead.

Expose options for handling tool dependencies etc when installing tool

Currently manage_tools install... always handles repository and tool dependencies, by setting the install_tool_dependencies and install_repository_dependencies arguments of install_repository_revision to True (see http://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#bioblend.galaxy.toolshed.ToolShedClient.install_repository_revision).

However these settings could be exposed via the CLI, to allow the user to control how these are handled.

Conda dependencies not installed with tool for Galaxy 17.05

When using e.g. install_tool on Galaxy 17.05, tools which don't define any explicit toolshed dependencies don't have any associated conda dependencies installed.

For example:

nebulizer install_tool local toolshed.g2.bx.psu.edu devteam bowtie2 8ccbdbe9a695

If the tool is installed via Galaxy's admin interface then the conda dependencies are installed okay.

(Note btw that it's not obvious how to force installation of the missing dependencies afterwards.)

Add "ping" command

Request to add a ping command which could be used to test whether a Galaxy server is alive and responding to requests, e.g.

$ nebulizer ping main
PING main (https://usegalaxy.org)
64 bytes from main (https://usegalaxy.org): time = 0.049 ms
...
^C

Looking at the UNIX ping program the following options might be useful to implement:

  • -c COUNT: sends COUNT requests
  • -i INTERVAL: frequency to send requests
  • -W TIMEOUT: how long to wait for a response from the server

Add "whoami" command

It might be useful to implement a whoami command to recover the user name from the API key.

The functionality might also be useful to check if the connection can be used for operations which require an associated user (e.g. data library uploads, see issue #25).

Allow 'update_tool' to operate on multiple repositories

Proposal to extend the update_tool command to allow it to operate on multiple repositories in a single invocation.

This could be in one or more of the following forms

  • Allow specifying multiple repositories e.g. nebulizer update_tool galaxy pjbriggs/trimmomatic devteam/fastqc
  • Allow wildcards in repository names e.g. nebulizer update_tool galaxy iuc/trinity_*
  • Add an option to update all repositories with newer versions e.g. nebulizer update_tool galaxy --all or nebulizer update_tool galaxy */*

manage_tools installed --list-tools doesn't correctly associate tools with repos

The manage_tools installed ... --list-tools doesn't correctly associate the tools with their parent repos, for example:

* weeder2   toolshed.g2.bx.psu.edu  pjbriggs    2:3c5f10f7dd40  Installed
- Weeder2   2.0.0   Motif discovery in sequences from coregulated genes of a single species
- Weeder2   2.0.1   Motif discovery in sequences from coregulated genes of a single species
^ weeder2   toolshed.g2.bx.psu.edu  pjbriggs    1:571cb77ab9e7  Installed
- Weeder2   2.0.0   Motif discovery in sequences from coregulated genes of a single species
- Weeder2   2.0.1   Motif discovery in sequences from coregulated genes of a single species
total 2

Here the 2.0.1 version is actually from the first repo/version, and the 2.0.0 version is from the second - the tools should only be listed once, and with the correct repo/version.

Standardise 'install_tool' behaviour

It's not clear that the behaviour of install_tool is entirely conistent, so some standardisation would be useful:

  • If no revision is specified then stop if any revision of a tool is already present
  • If a revision is specified then only stop if that revision isn't already installed
  • Implement better handling of multiple repositories (specifically when using the --file option, however could be extended to allow install_tool to install multiple tools in a single invocation).

Wait for tools with status 'New' when installing tool repositories

In the tools.install_tool function, tools with status New are not counted as currently installing, but tools can often wait in this state for some time while their dependencies are being installed.

To handle this, the function should be updated to also count New as an installing status.

Implement a test framework to test against Galaxy & toolshed instances

Implement some form of test framework that can be used with unittest, to allow testing of nebulizer components. Then use it to implement some tests.

What I imagine is something that will:

  • Clone Galaxy repo and set to a specific revision
  • Create and populate galaxy.ini/tool_shed.ini files suitable for testing with
  • Start up Galaxy/toolshed and indicate when they're ready to interact with
  • Shut them down at the end

Bonus points for:

  • Being able to run Galaxy using source code from elsewhere, without overwriting
  • Using a prebuilt SQLite database to speed up initialisation
  • Adding users, tools, categories etc for testing

Check status code of connection errors from Bioblend

Bioblend's ConnectionError class has two attributes:

  • status_code: the status code returned from the sever (e.g. 502)
  • body: the data returned by the server (e.g. nginx "service not available")

(See http://bioblend.readthedocs.io/en/latest/_modules/bioblend.html#ConnectionError)

It would be useful in most cases to handle these errors more gracefully by checking the status code and taking action accordingly.

List of HTTP status codes (with explanations) can be found at e.g. https://en.wikipedia.org/wiki/List_of_HTTP_status_codes

Fix ordering of tools from "manage_tools list"

The ordering of tools printed by the manage_tools list command could be improved to group together multiple versions of the same tool/shed/owner combination.

Here's an example of what can happen at the moment:

pal_finder          0.02.04.1   Microsatellite Analysis toolshed.g2.bx.psu.edu/pjbriggs/pal_finder
pal_finder          0.02.04.3   Developmental Tools XXXXXXXXX/toolshed/pjbriggs/pal_finder
pal_finder          0.02.04.2   Microsatellite Analysis toolshed.g2.bx.psu.edu/pjbriggs/pal_finder

Utilities to upload/download files from histories?

It would be nice to give users utilities that allowed them to pull a dataset or set of datasets down from Galaxy in order to work on it using the command line/cluster/whatever, and also give them an easy way to upload files without using the web interface or FTP server.

The API has options for this which are exposed in bioblend:

  • tools -> upload_to_history
  • datasets -> download_dataset

'install_tool': prompt user to confirm tool installation by default?

Proposal to add a prompt in the install_tool command for the user to confirm that they want to proceed with the installation (similar to the prompt already implemented for the update_tool and delete_tool commands), with a -y option to override this on the command line.

Consolidate and simplify tool/repository management commands

The tool/repository management commands are a bit of a mixed bag at the moment, there are several different commands for slightly different situations:

  • list_tools: lists tools that appear in the tool panel (both native and toolshed-installed)
  • list_installed_tools: lists toolshed-installed repositories in user-friendly style
  • list_repositories: lists toolshed-installed repositories in a format that can be fed into install_repositories
  • install_tool: installs a single repository from a toolshed
  • install_repositories: installs all the repositories from a file produced by list_repositories

There is also update_tool which updates a single toolshed-installed repository to the latest version.

It is suggested that these could be consolidated into a smaller set of commands (inspired by conda's install and list commands), e.g.:

  • install_tool -> install_tool
  • install_repositories -> install_tool --file FILE
  • list_installed_tools -> list_tools
  • list_repositories -> list_tools --export

Possibly also drop the functionality of the current list_tools command?

(The names of the new commands might have to be diffferent, to best reflect the terminology used within Galaxy itself.)

Add an interactive "shell" mode

Would be nice to have an interactive "shell" mode, e.g.

$ nebulizer localhost
list_users
...

which would preserve context when performing a number of related actions.

Add option to 'uninstall_tool' to prune older versions of tools

It might be useful to add an option to the uninstall_tool command to remove older versions of tools, i.e. "pruning" rather than completely uninstalling all revisions. It should be possible to specify the maximum number of older revisions to keep (e.g. last three).

For example something like:

nebulizer uninstall_tool GALAXY pjbriggs/trimmomatic --prune --keep=3

Implement fallback options to handling missing or deprecated versions when installing tools

This issue covers two situations when attempting to install or update a tool:

  • If the requested version is no longer installable then installation will fail (i.e. nebulizer install_tool will do nothing)
  • If the requested or latest version is deprecated in the toolshed then nebulizer install_tool will still attempt to install it, and nebulizer update_tool will attempt to update to it.

In the first case, it would be useful to be able to tell Nebulizer to install the most recent installable version of a tool instead in the event that the requested version is not available.

In the second case, it would make sense to treat deprecated tools as uninstallable and follow the fallback procedure above (i.e. go to the most recent installable revision), or else enable an option to allow installation of the deprecated tools (i.e. a switch to say that deprecated tools are installable).

Don't include tool repositories with no status when listing updateable tools

In the list_installed_tools command, the --updateable option currently includes repositories without any status (e.g. because no information could be fetched from the source toolshed for the tool).

For example you might see:

U tabular_to_fastq	toolshed.g2.bx.psu.edu	devteam	1:92034dcbb40a	Installed
U varscan_version_2	toolshed.g2.bx.psu.edu	devteam	1:44d514f3df8f	Installed
  weeder2	xxxxxxxxx.itservices.manchester.ac.uk/toolshed	pjbriggs	1:51d666bf37b1	Installed

In this case weeder2 shouldn't be listed in the results.

Prevent library upload if API key isn't associated with a real user account

Previously it seems that attempting to upload datasets to a data library using the API will fail if the API key being used isn't associated with a real user account (the case when the master API key is being used to access Galaxy).

In these cases Nebulizer should refuse to attempt the upload operation.

NB it's not clear if this limitation extends to the creation of libraries or folders themselves.

Check tool install status when user cancels installation

When cancelling a tool installation request (e.g. by doing ^C on nebulizer install_tool...), it is possible that Galaxy has already begun actioning the request. In this case although nebulizer has stopped, the tool installation will still be proceeding on the server - in which case there is no way to stop this through either the API or the Galaxy interface.

So: it would be useful if nebulizer could check and report the tool installation status on abort, so that the user at least is aware if the installation process is still running.

Using Click 7.0 changes subcommand names from e.g. "list_users" to "list-users"

Changes in the latest version of the click package (7.x) means that nebulizer subcommands which had underscores now automatically have these converted to dashes.

From https://click.palletsprojects.com/en/7.x/changelog/

Subcommands that are named by the function now automatically have the underscore replaced with
a dash. If you register a function named my_command it becomes my-command in the command
line interface.

Essentially this breaks nebulizer's CLI compared to the documentation.

Add commands to manipulate Galaxy groups

Bioblend exposes a number of API functions for manipulating groups in a Galaxy instance, which would be useful to add in nebulizer, see http://bioblend.readthedocs.io/en/latest/api_docs/galaxy/all.html#module-bioblend.galaxy.groups

Probably most useful for me would be:

  • create a new group
  • add or remove users from a group
  • show group members

(There are also functions for manipulating roles but I don't use these so much however they could be useful too.)

Update: additionally it would be good to have functionality for flagging up "empty" groups etc.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.