cwl-conditionals's Introduction

CWL conditionals test bed

Useful links

Steps for getting up an running

tl;dr

git clone https://github.com/kaushik-work/cwl-conditionals.git
chmod +x ./cwl-conditionals/setup.sh
./cwl-conditionals/setup.sh

If you don't have Python3 (callable as python3) you can replace the line in setup.sh to create a Python2 virtualenv.

Also see setup.sh

Create a new virtual environment for testing
Install Toil from kaushik-work/toil kaushik/conditionals/when branch
Install CWL tool pa/conditionals branch
Checkout the test scripts and test files
Run them with toil-cwl-runner

Manual steps (in a fresh virtualenv):

# If toil is not installed from clone, it complains about missing galaxy libs
git clone --branch kaushik/conditionals/when https://github.com/kaushik-work/toil.git
cd toil
pip install -e .[cwl]
toil --version  # should read 3.21.0a1
cd ..


git clone --branch pa/conditionals https://github.com/common-workflow-language/cwltool.git
cd cwltool
pip install -e .
cd ..

cd cwl-conditionals
chmod +x run-examples.sh
./run-examples.sh

Running Toil in macOS

You may need to do the following before running a task that uses Docker export TMPDIR=/tmp/docker_tmp from Toil

cwl-conditionals's People

Contributors

Stargazers

Watchers

cwl-conditionals's Issues

unable to run examples

$ ./run-examples.sh
mrc-tux 2019-11-15 13:12:24,314 MainThread INFO cwltool: Resolved 'workflows/baseline.cwl' to 'file:///home/michael/src/toil-conditionals/cwl-conditionals/workflows/baseline.cwl'
mrc-tux 2019-11-15 13:12:24,836 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (8).
mrc-tux 2019-11-15 13:12:24,836 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (67215048704).
mrc-tux 2019-11-15 13:12:24,836 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (66731671552).
mrc-tux 2019-11-15 13:12:24,861 MainThread INFO toil: Running Toil version 3.21.0a1-14db37153cf08d202f50e160b1531b35fe66ce4d.
INFO:toil.worker:Redirecting logging to /tmp/toil-a1823c84-73d5-46fb-b269-8716670f88c1-8f41ddb6e30e4bc0b9adde0837fcd6eb/tmpgtw5wlw2/worker_log.txt
mrc-tux 2019-11-15 13:12:25,226 MainThread INFO toil.leader: Issued job 'file:///home/michael/src/toil-conditionals/cwl-conditionals/tools/list-in.cwl' echo kind-file_home_michael_src_toil-conditionals_cwl-conditionals_tools_list-in.cwl/instance85e2j5gj with job batch system ID: 1 and cores: 1, disk: 3.0 G, and memory: 2.0 G
INFO:toil.worker:Redirecting logging to /tmp/toil-a1823c84-73d5-46fb-b269-8716670f88c1-8f41ddb6e30e4bc0b9adde0837fcd6eb/tmpc2c6jd5e/worker_log.txt
mrc-tux 2019-11-15 13:12:25,570 MainThread INFO toil.leader: Job ended: 'file:///home/michael/src/toil-conditionals/cwl-conditionals/tools/list-in.cwl' echo kind-file_home_michael_src_toil-conditionals_cwl-conditionals_tools_list-in.cwl/instance85e2j5gj
INFO:toil.worker:Redirecting logging to /tmp/toil-a1823c84-73d5-46fb-b269-8716670f88c1-8f41ddb6e30e4bc0b9adde0837fcd6eb/tmpajqghi_q/worker_log.txt
INFO:toil.worker:Redirecting logging to /tmp/toil-a1823c84-73d5-46fb-b269-8716670f88c1-8f41ddb6e30e4bc0b9adde0837fcd6eb/tmp4p2lt505/worker_log.txt
mrc-tux 2019-11-15 13:12:27,880 MainThread INFO toil.leader: Finished toil run successfully.
{
    "out1": [
        "a_[\"A\", \"B\"]_b",
        "A"
    ]
}mrc-tux 2019-11-15 13:12:27,891 MainThread INFO toil.common: Successfully deleted the job store: FileJobStore(/tmp/tmpvyfn0ofc)
mrc-tux 2019-11-15 13:12:28,237 MainThread INFO cwltool: Resolved 'workflows/pattern0-1.cwl' to 'file:///home/michael/src/toil-conditionals/cwl-conditionals/workflows/pattern0-1.cwl'
Traceback (most recent call last):
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/ref_resolver.py", line 168, in fetch_text
    resp.raise_for_status()
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://w3id.org/cwl/CommonWorkflowLanguage.yml

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/ref_resolver.py", line 722, in resolve_ref
    doc = self.fetch(doc_url, inject_ids=(not mixin))
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/ref_resolver.py", line 1183, in fetch
    text = self.fetch_text(url)
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/ref_resolver.py", line 170, in fetch_text
    raise_from(RuntimeError(url, e), e)
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/future/utils/__init__.py", line 403, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
RuntimeError: ('https://w3id.org/cwl/CommonWorkflowLanguage.yml', HTTPError('404 Client Error: Not Found for url: https://w3id.org/cwl/CommonWorkflowLanguage.yml'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/michael/src/toil-conditionals/env3.7/bin/toil-cwl-runner", line 8, in <module>
    sys.exit(main())
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/toil/cwl/cwltoil.py", line 1309, in main
    loading_context, uri = cwltool.load_tool.resolve_and_validate_document(loading_context, workflowobj, uri)
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/cwltool/load_tool.py", line 272, in resolve_and_validate_document
    process.get_schema(cwlVersion)[:2]
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/cwltool/process.py", line 190, in get_schema
    "https://w3id.org/cwl/CommonWorkflowLanguage.yml", cache=cache)
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/schema.py", line 245, in load_schema
    schema_doc, schema_metadata = metaschema_loader.resolve_ref(schema_ref, "")
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/ref_resolver.py", line 722, in resolve_ref
    doc = self.fetch(doc_url, inject_ids=(not mixin))
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/schema_salad/sourceline.py", line 298, in __exit__
    raise_from(self.makeError(six.text_type(exc_value)), exc_value)
  File "/home/michael/src/toil-conditionals/env3.7/lib/python3.7/site-packages/future/utils/__init__.py", line 403, in raise_from
    exec(execstr, myglobals, mylocals)
  File "<string>", line 1, in <module>
RuntimeError: ('https://w3id.org/cwl/CommonWorkflowLanguage.yml', HTTPError('404 Client Error: Not Found for url: https://w3id.org/cwl/CommonWorkflowLanguage.yml'))