Resume function in the pipeline

[encode-dcc]

Resume function in the pipeline

[Cristinex]

Cromwell supports "call-caching" and I am sure that is what you are looking for. It skips tasks that are already done based on docker hash, md5sum hash of input files and code lines in commands { } block. You may need to install a MySQL server and run a cromwell server with it and submit jobs via REST API.

But there are some limitations. Any change in a docker container (that is... any single update/bug-fix on the pipeline) will make you lose resumability. So you will not find it very useful.

We don't have an official tutorial or README for this. But I think the following documnets will be helpful.

• https://github.com/ENCODE-DCC/atac-seq-pipeline/blob/master/docs/tutorial_sherlock.md#running-multiple-pipelines-with-cromwell-server-mode
• https://github.com/ENCODE-DCC/atac-seq-pipeline/tree/master/test#how-to-run-a-cromwell-server-on-gc

I think the documents maybe helpful. However, the web page links you posted are now invalid. Could you update links to the document, please?

[leepc12]

Cromwell supports "call-caching" and I am sure that is what you are looking for. It skips tasks that are already done based on docker hash, md5sum hash of input files and code lines in commands { } block. You may need to install a MySQL server and run a cromwell server with it and submit jobs via REST API.

But there are some limitations. Any change in a docker container (that is... any single update/bug-fix on the pipeline) will make you lose resumability. So you will not find it very useful.

We don't have an official tutorial or README for this. But I think the following documnets will be helpful.

• https://github.com/ENCODE-DCC/atac-seq-pipeline/blob/master/docs/tutorial_sherlock.md#running-multiple-pipelines-with-cromwell-server-mode
• https://github.com/ENCODE-DCC/atac-seq-pipeline/tree/master/test#how-to-run-a-cromwell-server-on-gc

[shanmukhasampath]

Hi Jin,

Thank you very much for the documentation. I will look into it.

[leepc12]

My comment on this issue was written on 2018. It's outdated.

Please take a look at these docs for SLURM and GC:
https://github.com/ENCODE-DCC/caper#running-pipelines-on-slurm-clusters
https://github.com/ENCODE-DCC/caper/tree/master/scripts/gcp_caper_server

MySQL is no longer needed for call-caching.
File-based DB (db=file in Caper's conf file ~/.caper/default.conf) should work fine for a small number of workflows.