More generalized form of Parallelization than Splitting by Chromosome

[ssadedin]

More generalized form of Parallelization than Splitting by Chromosome

[lonsbio]

From [email protected] on 2014-08-12T01:42:16Z

I assume what you are describing here is generalizing it where any prefix could be put in, so

part(1..10) or part("A","B","C") or section(1..10), etc.

Will produce jobs with variables part1,part2,... or partA, partB, ..., or section1,section2, section3...

That's a pretty good start. I would suggest that you consider also how to pass in pure numbers to each job with no prefix for the numbers, as well as pairs of numbers. I have used a number of tools that take a range in a format like:

myscript -r begin,end

So there are actually two numbers to update for each job. Obviously you can just wrap the job in a script that interprets a variable like part1 into the correct begin, end indices, but it'd be nice if there were a way to do it in a bpipe script.

I'm not sure how to do this syntactically that would be easy enough to parse. If parsing were no barrier, maybe something like:

Bpipe.run{
(0..500).each{begin->
end = begin + 1000
[my_parallel_task]
}

where begin and end become implicit variables for each instance of the parallel task.