Unknown intermediate files in workflow

[cmkobel]

Unknown intermediate files in workflow

[dansondergaard]

I have a workflow where I extract all genes from a number of genomes, and then perform a series of statistics on these individual files. This means that when I start the workflow I don't know how many and what the names of these files will be.

The way gwf works, it needs to know about all input/output files to create the dependency graph, which happens before anything is actually run. So no, there's no nice solution.

I can't find a way to have gwf dynamically parse the individual gene files from the beginning. This means that I need to restart the workflow for each "checkpoint" where a number of intermediate files are generated.

From this it sounds like you actually do know which files you're going to end up with? Do you have files that specify the genes in the genomes already? If you do, it should be trivial to parse these in your workflow file and create the appropriate targets.

Cheers,
Dan

[cmkobel]

Hi Dan.

Thanks for your quick answer.

From this it sounds like you actually do know which files you're going to end up with? Do you have files that specify the genes in the genomes already? If you do, it should be trivial to parse these in your workflow file and create the appropriate targets.

The names of the genes in the genomes stem from a genome annotation (Prokka) which is part of the pipeline. So only if I pre-annotate can I do what you suggest.

Best, Carl.