pveber / bistro Goto Github PK
View Code? Open in Web Editor NEWA library to build and execute typed scientific workflows
License: Other
A library to build and execute typed scientific workflows
License: Other
Bistro functions may have arguments that are not workflows. In that case, the workflow's id should depend also on the value of these arguments, which cannot be expressed at the moment. Let's call these arguments parameters.
... that shows they've been run from bistro.
Currently all commands executed through docker are executed as root (in the container), which has (at least) two drawbacks:
One solution would be to execute command through docker after changing user in the container, choosing the same uid than the (bistro) process owner.
Looking at content of files, looks like software is GPL.
Would be nice to get in README, or in a LICENSE file, the license of the software to inform users without needing to dig in files
:-)
https://coreos.com/rkt/
https://coreos.com/rkt/docs/latest/rkt-vs-other-projects.html
Expected benefits:
Singularity offers ro or rw mounts, with the syntax:
singularity shell --mount /host:/guest:ro ...
Would be nice to use for cache and inputs.
It would be nice to have a mechanism to remove intermediary targets during the execution of the workflow, to save space in the cache.
Would be convenient to have a mechanism general allowing to inject run-time values in workflow scripts. Scripts can then be slightly customized while not changing their ID. Variables like np
and mem
could be reimplemented this way, and that would be a nice way to introduce the name of the currently run executable (to implement custom procedures in a portable way).
Upstreaming this from Nixpkgs:
# File "ppx/bistro_script.ml", line 201, characters 18-41:
# 201 | let e = Parser.parse_expression Lexer.token (Lexing.from_string txt) in
# ^^^^^^^^^^^^^^^^^^^^^^^
# Alert deprecated: module Ppxlib.Parser
# Accessing this module directly is deprecated, use Ocaml_common.Parser instead
# File "ppx/bistro_script.ml", line 201, characters 18-41:
# 201 | let e = Parser.parse_expression Lexer.token (Lexing.from_string txt) in
# ^^^^^^^^^^^^^^^^^^^^^^^
# Error: Unbound value Parser.parse_expression
Indeed Nixpkgs simply patches Parser.parse_expression
to Ocaml_common.Parser.parse_expression
here.
Suggestion for a phylogenetic pipeline by @Boussau:
Items of the same level are alternatives
clustering:
alignement:
gene phylogeny:
species phylogeny:
It seems mounts as they are done in engine/Task
do not work on MacOS X, probably because Docker runs in a virtual machine, which doesn't host the files
for instance getting inspiration from https://pilsniak.com/how-to-install-docker-on-mac-os-using-brew/
File "ppx/bistro_script.ml", line 205, characters 58-59:
File "ppx/bistro_script.ml", line 205, characters 58-59:
205 | (new ast_translation loc'.loc_start)#expression e
205 | (new ast_translation loc'.loc_start)#expression e
^
^
Error: This expression has type Parsetree.expression
Error: This expression has type Parsetree.expression
but an expression was expected of type
but an expression was expected of type
Ppxlib.Ast.expression = Astlib.Ast_412.Parsetree.expression
Ppxlib.Ast.expression = Astlib.Ast_412.Parsetree.expression
For large workflows with many shared sub-workflows, the serialization via s-expressions produces a very large representation (exponential?). This hurts in particular when saving a workflow description in the database (currently only in the statistics table).
use_docker
option in Bistro_app
)When several tasks require significant bandwidth, it would be better to run them one after the other, to avoid too much idling. One solution would be to have a flag to say that a workflow consumes mostly network bandwidth and add network bandwidth as a limited resource.
The repository generated by Bistro_app.with_backend
is populated with absolute links to the cache, but this prevents a move of the containing directory. It would be better to switch to relative links.
Alternative to htseq-count
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.