Comments (18)
@johanneskoester would you be interested in this?
from snakemake-wrappers.
As cited above, a container URL in the wrapper meta yaml should be enough to configure it.
@johanneskoester are you suggesting that if the container URI is in the meta.yaml
then there is no need for envornment.yaml
? Or have both and if the user doesn't specify --use-singularity
then we default to the conda env?
@vsoch I'll provide an example from bwa mem
.
meta.yaml
name: "bwa mem"
description: Map reads using bwa mem, with optional sorting using
samtools or picard.
authors:
- Johannes Köster
- Julian de Ruiter
So I guess we could just add something akin to snakemake
where we say container: <URI>
?
from snakemake-wrappers.
I was thinking it might be good to provide both a conda environment (if applicable) and a container for the wrapper. Then, we decide which environment to use based on the snakemake CLI options
# use wrapper's conda env (if there is one)
snakemake --use-conda
# use wrapper's container env (if there is one)
snakemake --use-singularity
# use wrapper's container env. if no container env, use conda env
snakemake --use-singularity --use-conda
This way the user doesn't have to think about additional parameters within the snakefile and it is all controlled with existing CLI options that are already required for using wrappers and containers.
from snakemake-wrappers.
@vsoch you are impressive!!
I have some questions/comments but will move the discussion to the PR
from snakemake-wrappers.
I am transferring this as I would like to reignite the conversation.
This issue is the sole issue preventing me from using wrappers at the moment (and some people in my group).
Would you like some help with this @johanneskoester (that is if you think this feature should be added)? If so, would you be able to point me towards where would be the best place for me to start investigating a solution?
from snakemake-wrappers.
Thanks Mike
I am still very interested in this feature.
I also like to use wrapper locally (see https://github.com/a-slide/NanoSnake). So I would say having the possibility to use a local singularity recipe instead of the conda YAML would be very cool.
from snakemake-wrappers.
I can offer to help as well! @mbhall88 and @a-slide in case you didn't notice at the top, please note #59 as well!
from snakemake-wrappers.
Great! Thanks @vsoch . And yes, I did see the message from Johannes - the poor guy is going to have so many notifications to come back to.
Did you want to have a chat to sort out how best to approach implementing this? What is the best way for communicating for you? (maybe we can continue this via DM on twitter? or anywhere else is fine)
from snakemake-wrappers.
I wouldn't be the right one to have an idea for best practices for implementation, I have worked on snakemake source code but not snakemake-wrappers source code. If you think you have enough experience to be that guide, then we can definitely give it a go and then get Johannes feeback! But otherwise, I think we should at least wait for @johanneskoester to come back and give some vision for how it's best done.
And reaching me - yes DM on Twitter works, as does any channel on Gitter.
from snakemake-wrappers.
Ok, fair point. We'll wait for Johannes.
from snakemake-wrappers.
Hi folks,
I'm back :-). Absolutely, I would be very happy to include this. As cited above, a container URL in the wrapper meta yaml should be enough to configure it. Then, one needs a function snakemake.wrapper.get_container_img(path, prefix=None)
, analog to snakemake.wrapper.get_conda_env. And then, you just need to use it analogously to get_conda_env in workflow.py here.
All in all, just a few lines of code I think. Which makes me even more sorry to not having done this myself already.
from snakemake-wrappers.
@mbhall88 do you have an example of the wrapper meta yaml so I could see what it looks like? I see that the dag also pulls singularity containers and I'm trying to figure out how those two are different.
from snakemake-wrappers.
@mbhall88 ah I think I understand now! So the wrapper would essentially have a singularity (via docker) image backend instead of using conda. @mbhall88 what about if a recipe would want to provide a conda and container option? Would we want to maintain separate wrappers, e.g.::
rule samtools_sort:
input:
"mapped/{sample}.bam"
output:
"mapped/{sample}.sorted.bam"
params:
"-m 4G"
threads: 8
wrapper:
"0.0.8/bio/samtools_sort_container"`
or even a different "container" namespace that mirrors the top level:
wrapper:
"0.0.8/containers/bio/samtools_sort"`
Or add some parameter to the recipe for the user to choose (and then have a default?). My worry with doing the first is that the user would expect the exact same outcome / performance between the container and conda runs, and I'm not sure we could promise that. On the other hand, maintaining them separately is also challenging. What do you think?
from snakemake-wrappers.
I like that too! And probably since most wrappers are conda now, we would default to that if a container isn't available. And this also means we would try our best to match versions, but of course it can't be absolutely guaranteed that the container and conda install would be exactly the same.
One more question for you @mbhall88 before we look at code - I'm concerned about mapping containers to snakemake-wrappers. For example, if you look at the biocontainers bwa Dockerfile, it's only installing bwa, but the wrapper that you linked has bwa, samtools, and picard. We should decide on some strategy for mapping containers to wrappers.
- we could use biocontainers, and only map those that are absolute matches (I'm not sure I like this approach because the versions would move at different times and many wrappers don't have an absolute match)
- we could have an automated build to docker hub,perhaps a Snakemake wrappers organization, that always has the correct versions and software (this is my preference). So a wrapper here would have a template for an automated build (we could easily use GitHub actions) and the build would also be tested when a new wrapper version is created. I'm (personally) more of a fan of using Quay.io because we could have repository specific robot tokens, as opposed to docker hub which has tokens only for an entire user account. But we could also try setting up a hook (automated build) if docker hub is preferred. What do you think?
from snakemake-wrappers.
I'm concerned about mapping containers to snakemake-wrappers. For example, if you look at the biocontainers bwa Dockerfile, it's only installing bwa, but the wrapper that you linked has bwa, samtools, and picard. We should decide on some strategy for mapping containers to wrappers.
Yes, you're right. This is a pain point.
I like your second option.
One question I have about this is are you recommending we have a Dockerfile with each wrapper? Or it auto-builds an image based on the packages and versions in the environment.yaml?
from snakemake-wrappers.
Sort of both - we would have a Dockerfile that uses the environment.yml to install stuffs. That way, they are as close to identical as possible! The one issue I see for setting up automated builds is the lack of modularity in the single repo. Ideally we would have one automated build per repo without some complicated logic for the trigger and settings. I was thinking we should slowly port over to a Snakemake-wrappers org, but of course we need feedback from @johanneskoester first! But if y’all like the idea I can take a first shot at a modular repo with an automated build and then work on a Snakemake PR to add the functionality.
from snakemake-wrappers.
@mbhall88 one more question - does this approach assume we need to read in the meta.yaml file to discover the container? Or did you have something else in mind? I did a grep of snakemake and (as far as I can tell) it's not read in anywhere, because largely we don't need it for the run.
from snakemake-wrappers.
hey @mbhall88 - happy Saturday! I got us started on something to get this ball rolling, see snakemake/snakemake#532. Note that the implementation will require changes to the wrappers repository here, and I've detailed what we need to discuss in the PR description.
from snakemake-wrappers.
Related Issues (20)
- The ensembl-annotation uses the wrong download URL HOT 3
- Error while testing bbduk HOT 1
- Salmon Quant wrapper throwing segmentation fault HOT 1
- MACS2: support --extsize with fragment size input file
- bwa-memx_index meme wrapper expects specific file name? HOT 3
- Possible bowtie2 align bug HOT 4
- Generic subfolder? HOT 1
- STAR wrapper allowing different output names HOT 3
- CI: autobump workflow never finishes HOT 2
- GFFREAD wrapper produces gff output when fasta file requested HOT 1
- building the docs on a fork doesn't work HOT 2
- FastQC resource allocation not working as expected - get_mem fix? HOT 3
- MultiQC wrapper env issue with imp HOT 1
- Wrappers for Google Batch HOT 3
- ModuleNotFoundError: No module named 'snakemake_wrapper_utils HOT 10
- `bcftools reheader` no longer works with map-style `sample.tsv`
- `bcftools merge`: not able to declare dependency on index files HOT 1
- `deepvariant`: conda environment not installable HOT 3
- change to earlier docs versions on readthedocs HOT 4
- SAMTOOLS FASTQ INTERLEAVED outputs to `log` instead of `output` HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from snakemake-wrappers.