duke-gcb / calrissian Goto Github PK
View Code? Open in Web Editor NEWCWL on Kubernetes
Home Page: https://duke-gcb.github.io/calrissian/
License: MIT License
CWL on Kubernetes
Home Page: https://duke-gcb.github.io/calrissian/
License: MIT License
Using the docker image dukegcb/calrissian:0.3.1
I received the following error when running the exomeseq-gatk4-preprocessing.cwl workflow.
Traceback (most recent call last):
File "/usr/local/bin/calrissian", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/calrissian/main.py", line 83, inmain
write_report(parsed_args.usage_report)
File "/usr/local/lib/python3.6/site-packages/calrissian/report.py", line 343,in write_report
json.dump(Reporter.get_report().to_dict(), f, indent=4, default=default_serializer)
File "/usr/local/lib/python3.6/site-packages/calrissian/report.py", line 290,in to_dict
result = super(TimelineReport, self).to_dict()
File "/usr/local/lib/python3.6/site-packages/calrissian/report.py", line 39, in to_dict
result['elapsed_hours'] = self.elapsed_hours()
File "/usr/local/lib/python3.6/site-packages/calrissian/report.py", line 35, in elapsed_hours
return self.elapsed_seconds() / SECONDS_PER_HOUR
File "/usr/local/lib/python3.6/site-packages/calrissian/report.py", line 27, in elapsed_seconds
delta = self.finish_time - self.start_time
TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'
Command was:
calrissian --default-container debian:stretch-slim --tmp-outdir-prefix /bespin/tmpout/ \
--outdir /bespin/output-data/results/ --max-ram 120G --max-cores 52 \
--usage-report /bespin/output-data/job-94-jpb67-resource-usage.json \
/bespin/job-data/workflow/exomeseq-gatk4-preprocessing.cwl /bespin/job-data/job-order.json
Currently only supporting hostPath volumes, but will need support for read-write many volumes such as GlusterFS.
Currently the finish method always reports 'success'
:
Lines 209 to 214 in b4526af
This should actually inspect the result of the kubernetes job and report failure
Current executor is a thin subclass of MultithreadedJobExecutor, so it schedules/accounts jobs based on the local machine's CPU and RAM.
Obviously the local machine's resources should not be used for jobs in a cluster
Additionally, the jobs created should be able to include the requested resources for accurate scheduling.
https://github.com/common-workflow-language/common-workflow-language is the canonical location for the CWL v1.0 conformance tests
We should be able to re-run workflows with the same version of calrissian.
To support multiple k8s clouds we will need a central location for storing calrissian images.
For example dockerhub.
follow_logs
was added in #59, and calls read_namespaced_pod_log()
with follow=True
.
In my tests of the GATK4 preprocessing workflow, I can see that the generator returned by stream()
finishes while the pod is still running.
logs from calrissian job:
$ oc logs --timestamps=True job/calrissian-gatk4-preprocessing |grep trim-pod
2019-03-14T18:52:27.643057398Z [trim-pod-klhihztw] 2019-03-14T18:52:28.377254055Z >>> Now performing quality (cutoff 20) and adapter trimming in a single pass for the adapter sequence: 'AGATCGGAAGAGC' from file /var/lib/cwl/stg5811d013-3646-40db-9308-efdf548c1776/SA05051-R2.fastq.gz <<<
2019-03-14T18:58:30.490922326Z [trim-pod-klhihztw] 2019-03-14T18:58:31.219549578Z 10000000 sequences processed
2019-03-14T19:04:36.760504315Z [trim-pod-klhihztw] 2019-03-14T19:04:37.48981895Z 20000000 sequences processed
2019-03-14T19:10:50.103854762Z [trim-pod-klhihztw] 2019-03-14T19:10:50.83159839Z 30000000 sequences processed
2019-03-14T19:17:27.807121329Z [trim-pod-klhihztw] 2019-03-14T19:17:28.540569252Z 40000000 sequences processed
2019-03-14T19:22:32.967763539Z [trim-pod-klhihztw] follow_logs end
logs directly from the pod
$ oc logs --timestamps trim-pod-klhihztw
2019-03-14T18:52:28.377254055Z >>> Now performing quality (cutoff 20) and adapter trimming in a single pass for the adapter sequence: 'AGATCGGAAGAGC' from file /var/lib/cwl/stg5811d013-3646-40db-9308-efdf548c1776/SA05051-R2.fastq.gz <<<
2019-03-14T18:58:31.219549578Z 10000000 sequences processed
2019-03-14T19:04:37.48981895Z 20000000 sequences processed
2019-03-14T19:10:50.83159839Z 30000000 sequences processed
2019-03-14T19:17:28.540569252Z 40000000 sequences processed
2019-03-14T19:22:33.692899536Z This is cutadapt 1.14 with Python 3.5.2
2019-03-14T19:22:33.693016208Z Command line parameters: -f fastq -e 0.1 -q 20 -O 1 -a AGATCGGAAGAGC /var/lib/cwl/stg5811d013-3646-40db-9308-efdf548c1776/SA05051-R2.fastq.gz
2019-03-14T19:22:33.693031811Z Trimming 1 adapter with at most 10.0% errors in single-end mode
The pod was running for about an hour when follow_logs
exited:
2019-03-14T18:19:44.408873685Z k8s pod 'trim-pod-klhihztw' started
2019-03-14T18:19:49.311305246Z [trim-pod-klhihztw] follow_logs start
...
2019-03-14T19:22:32.967763539Z [trim-pod-klhihztw] follow_logs end
When running within google cloud k8s engine the pods created by calrissian to run various steps get stuck in the ContainerCreating state. The Events for these pods show
AttachVolume.Attach failed for volume "<...gce pd name...> " : googleapi: Error 400: RESOURCE_IN_USE_BY_ANOTHER_RESOURCE - The disk resource '...gce pd name...' is already being used by '..k8s node..'
I think this might be fixed by declaring the persistentVolumeClaim as readOnly as described here:
kubernetes/kubernetes#67313 (comment)
Write unit tests and enable CI
Test [68/128] Test that second expression in concatenated valueFrom is not ignored
Test 68 failed: /usr/local/bin/calrissian --max-ram 8G --max-cores 4 --default-container debian:stretch-slim --outdir=/output/tmps26ft6ts --quiet v1.0/vf-concat.cwl v1.0/empty.json
Test that second expression in concatenated valueFrom is not ignored
Compare failure expected: {
"out": "a string\n"
}
got: {
"out": "\n"
}
caused by: failed comparison for key 'out': expected: "a string\n"
got: "\n"
Since all CommandLineTools in this engine need to run in a container, this class should ensure that the job runner uses a default docker image, as reana does:
I wrote unit tests in #13 but skipped over this method. We don't plan to use this feature right now, and the code written is already tested from CWL. Return to this when we can test the actual feature rather than blindly writing a unit test in the mold of the existing code.
After a recent attempt to run a larger job with some failures, the allocation counts got to be negative:
run_job: Allocated 9/120.0 CPU cores, 38000/257698.03776 MB RAM
wait_for_next_completion: Allocated 9/120.0 CPU cores, 38000/257698.03776 MB RAM
[step map_13] completed success
shutil.rmtree(/tmp/skha43nv, True)
shutil.rmtree(/tmp/fl5b_1h3, True)
run_job: Allocated -3/120.0 CPU cores, 22000/257698.03776 MB RAM
wait_for_next_completion: Allocated -3/120.0 CPU cores, 22000/257698.03776 MB RAM
[step sort_11] completed success
shutil.rmtree(/tmp/kxgp070z, True)
shutil.rmtree(/tmp/swxj439s, True)
run_job: Allocated -4/120.0 CPU cores, 17000/257698.03776 MB RAM
wait_for_next_completion: Allocated -4/120.0 CPU cores, 17000/257698.03776 MB RAM
[step sort_15] completed success
shutil.rmtree(/tmp/d3iyf495, True)
shutil.rmtree(/tmp/sj80vchm, True)
run_job: Allocated -5/120.0 CPU cores, 12000/257698.03776 MB RAM
wait_for_next_completion: Allocated -5/120.0 CPU cores, 12000/257698.03776 MB RAM
[step sort_14] completed success
shutil.rmtree(/tmp/0l7a2ou2, True)
shutil.rmtree(/tmp/lgfgj73u, True)
run_job: Allocated -6/120.0 CPU cores, 7000/257698.03776 MB RAM
wait_for_next_completion: Allocated -6/120.0 CPU cores, 7000/257698.03776 MB RAM
[step sort] completed success
shutil.rmtree(/tmp/xxvilo3g, True)
shutil.rmtree(/tmp/ttdwcuqk, True)
run_job: Allocated -7/120.0 CPU cores, 2000/257698.03776 MB RAM
wait_for_next_completion: Allocated -7/120.0 CPU cores, 2000/257698.03776 MB RAM
[step sort_3] completed success
shutil.rmtree(/tmp/8r8f5oxi, True)
shutil.rmtree(/tmp/a8f491ya, True)
run_job: Allocated -8/120.0 CPU cores, -3000/257698.03776 MB RAM
wait_for_next_completion: Allocated -8/120.0 CPU cores, -3000/257698.03776 MB RAM
[step mark_duplicates_7] completed success
shutil.rmtree(/tmp/xl6e42es, True)
shutil.rmtree(/tmp/1b5jzgi5, True)
run_job: Allocated -9/120.0 CPU cores, -8000/257698.03776 MB RAM
wait_for_next_completion: Allocated -9/120.0 CPU cores, -8000/257698.03776 MB RAM
[step map_6] completed success
shutil.rmtree(/tmp/yzgwopzd, True)
shutil.rmtree(/tmp/fc6k6ybw, True)
run_job: Allocated -21/120.0 CPU cores, -24000/257698.03776 MB RAM
wait_for_next_completion: Allocated -21/120.0 CPU cores, -24000/257698.03776 MB RAM
[step map_23] completed success
shutil.rmtree(/tmp/4w6pygny, True)
shutil.rmtree(/tmp/ra3yze12, True)
run_job: Allocated -22/120.0 CPU cores, -29000/257698.03776 MB RAM
Workflow cannot make any more progress.
Final process status is permanentFail
When running in Docker under openshift, the following error appears at the start of a calrissian run:
Could not load extension schema https://schema.org/docs/schema_org_rdfa.html: ('https://schema.org/docs/schema_org_rdfa.html', LockFailed('failed to create /.cache/salad/b/2/8/9/0/calrissian-gatk4-preprocessing-pkg-7rxg9-55dae700.13435938927651500254',))
Could not load extension schema https://schema.org/docs/schema_org_rdfa.html: ('https://schema.org/docs/schema_org_rdfa.html', LockFailed('failed to create /.cache/salad/b/2/8/9/0/calrissian-gatk4-preprocessing-pkg-7rxg9-55dae700.13435938927651500254',))
I suspect that schema-salad's home-dir lookup for caching is involved, and setting an environment variable on the pod would fix it.
Provide a mechanism for installing calrissian into a different project/namespace. We could conceivably use openshift templates here, helm, or a simple sed script.
On the cluster I was testing the
calrissian
namespace was already taken. I created calrissian2 but had to change namespace a bunch of places.
Originally posted by @johnbradley in #16
A single volume may be mounted twice, but listing the volume itself twice causes the container to hang.
Succeeds:
apiVersion: batch/v1
kind: Job
metadata:
name: single-mount-twice
spec:
template:
spec:
containers:
- args: [date > /mount-double/double.txt]
command: [/bin/sh, -c]
image: debian:stretch-slim
name: single-mount-twice
volumeMounts:
- {mountPath: /mount-single, name: mount-single, subPath: mount-single}
- {mountPath: /mount-double, name: mount-single, subPath: mount-double}
restartPolicy: Never
volumes:
- name: mount-single
persistentVolumeClaim:
claimName: multimount
Fails:
apiVersion: batch/v1
kind: Job
metadata:
name: double-mount
spec:
template:
spec:
containers:
- args: [date > /mount-double/double.txt]
command: [/bin/sh, -c]
image: debian:stretch-slim
name: double-mount
volumeMounts:
- {mountPath: /mount-single, name: mount-single, subPath: mount-single}
- {mountPath: /mount-double, name: mount-double, subPath: mount-double}
restartPolicy: Never
volumes:
- name: mount-single
persistentVolumeClaim:
claimName: multimount
- name: mount-double
persistentVolumeClaim:
claimName: multimount
The KubernetesJobBuilder.container_args method builds up a simple command-line array connecting stdout, stdin, and stderr.
This method should address the same concerns as cwltool's JobBase._execute, doing the following:
More context in #30
In #62, I changed the behavior for each container to get a local emptyDir
volume mounted as /tmp
. This means that the calrissian/cwltool generated tmpdir
provided to the CalrissianCommandLineJob object usually goes unused and does not need to be located on a persistent volume. Prior to this change, a job's tmpdir would be a subdirectory inside of --tmpdir-prefix
, so we required that that be on a persistent volume that could be mounted by the container.
After the change, that behavior no longer applies. The /tmp
directory inside a container is local to that container, and should improve performance (since /tmp is no longer a network share)
However, there are still cases that would require a --tmpdir-prefix
to be provided to calrissian. Specifically, tools that use the InitialWorkDirRequirement with writable files or directories. This feature is implemented in calrissian by making a copy of the original required file into the job's tmpdir
(from the calrissian python process). So, that location must be a place that calrissian can write it and the container can modify it. I assume tmpdir
was chosen since this this writable file is temporary and isolated:
If true, the file or directory must be writable by the tool. Changes to the file or directory must be isolated and not visible by any other CommandLineTool process. This may be implemented by making a copy of the original file or directory.
Test [123/128] Test that expression engine does not fail to evaluate reference to self with unprovided input
Final process status is permanentFail
Test 123 failed: /usr/local/bin/calrissian --max-ram 8G --max-cores 4 --default-container debian:stretch-slim --outdir=/output/tmp4xzk7g0w --quiet v1.0/stage-unprovided-file.cwl v1.0/empty.json
Test that expression engine does not fail to evaluate reference to self with unprovided input
Returned non-zero
Currently a JSON object is printed to stdout which contains details about the files created in the job.
This data can be see in the logs but it is mixed with stderr.
These two data(stdout and stderr) should be persisted is some way outside of the job logs.
Create a setup.py that provides a command-line runner script
Calrissian will report CPU and memory usage. Storage is another important resource, and we have an issue Duke-GCB/lando#141 to better estimate these volumes.
Let's add a field to the reports that records the size of files generated at each step so that we can better size the intermediate (tmpout) volume, and perhaps /tmp too?
Our job runner should be able to report how long each step ran for and what resources it used. This will be critical for accounting but could also be used for real-time progress reporting.
@johnbradley and i discussed recently how to accomplish this, and I suggested it would be best addressed by calrissian
Some thoughts:
Using calrissian from image dukegcb/calrissian:0.2.1
I received the following error:
Workflow error, try again with --debug for more information:
(422)
Reason: Unprocessable Entity
HTTP response headers: HTTPHeaderDict({'Cache-Control': 'no-store', 'Content-Type':
'application/json', 'Date': 'Thu, 28 Feb 2019 19:08:52 GMT', 'Content-Length': '888'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":
"Pod \"pair-wc-packed.cwl-pod-xftmzyvq\" is invalid: spec.containers[0].name: Invalid value:
\"pair-wc-packed.cwl-container\": a DNS-1123 label must consist of lower case alphanumeric
characters or '-', and must start andend with an alphanumeric character (e.g. 'my-name', or
'123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')","reason":"Invalid",
"details":{"name":"pair-wc-packed.cwl-pod-xftmzyvq","kind":"Pod","causes":
[{"reason":"FieldValueInvalid","message":"Invalid value: \"pair-wc-packed.cwl-container\":
a DNS-1123 label must consist of lower case alphanumeric characters or '-', and must start and
end with an alphanumeric character (e.g. 'my-name', or '123-abc', regex used for
validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?')","field":"spec.containers[0].name"}]},"code":422}
I was using this workflow: https://raw.githubusercontent.com/johnbradley/toyworkflow/master/pair-wc-packed.cwl
It would be nice to be able to query for pods that are part of a workflow by a label/selector.
Add an argument or env var to provide as a prefix or similar, so that we can annotate job IDs and steps as done in lando:
Related to #38, but deleting a Calrissian job will not automatically delete the pods it submitted to the API.
We should clean these up when the process is killed. Suggestion: catch SIGTERM and delete any submitted pods.
Currently, /tmp
in each container is mounted from a subPath of a shared PV. So each container would get its own /tmp
but that would be mounted over a network. As I mentioned in #5 (comment), this is probably counter to what we want and should re-consider.
I suggest in #5 (comment) that tmp
should simply be mounted as a kubernetes emptyDir
.
Got workflow error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cwltool/executors.py", line 264, in runner
job.run(runtime_context)
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 553, in run
self.check_requirements()
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 390, in check_requirements
raise UnsupportedRequirement('Error: feature {}.{} is not supported'.format(feature, field))
cwltool.errors.UnsupportedRequirement: Error: feature DockerRequirement.dockerOutputDirectory is not supported
Workflow or tool uses unsupported feature:
Error: feature DockerRequirement.dockerOutputDirectory is not supported
In attempts to run exomeseq-gatk4-preprocessing, steps failed because multiple pods were submitted with the same name.
Though these names are based on the internal CWL job name, that's not enough to make them unique even in a single workflow job.
I tried calrissian --version
and it just printed help. I expected to see version info like cwltool
does.
Originally posted by @johnbradley in #42
Currently, the job builder assumes 4 specific PVCs and their mount points
Lines 39 to 44 in b4526af
These could either be dynamically looked up through the k8s API, or provided in a ConfigMap
Not terribly urgent if we continue the convention, but it is a kludge.
Calrissian currently has an integer --max-ram
flag that specifies the
Maximum amount of RAM in MB to use
It would be convenient to specify memory sizing in the same way kubernetes does
So for specifying larger sizes users could just do something like --max-ram 10Gi
This would help with Duke-GCB/lando#158
When an input data persistent volume is mounted with a trailing slash this causes files to be incorrectly mounted into the pods created by calrissian.
This results in the child pods having an error:
'terminated': {'container_id': 'docker://<someid>',
--
| 'exit_code': 1,
...
| 'message': None,
| 'reason': 'Error',
...
If you change this line in CalrissianJob-revsort.yaml to end in a slash:
- mountPath: /calrissian/input-data/
This line seems to be assuming there is an extra /
after the mountPath
:
Line 99 in 347f902
This results in incorrect files being mounted for example hello1.txt
:
volumeMounts:
--
...
| - {mountPath: /var/lib/cwl/stge5759f63-f4d0-401b-9e06-d7efaf517e87/hello1.txt,
| name: calrissian-input-data, readOnly: true, subPath: ello1.txt}
When running now with openshift metrics enabled, I notice the memory usage very high in calrissian. It seems to increase by several GB between running pods, and stays level during job execution
Other notes:
Also the calrissian job pod is using 15GB. It started using 4GB, then doubled to 8GB, and now is using 16GB
- top says its all under calrissian process and not nodejs or other stuff
- currently using about 20% CPU. it might be checksumming but slowly
The patched version of PyYAML has been released, but requirements.txt still specifies 3.13.
This requirement is a result of pip install kubernetes
in 5f343ad, so we'll need to upgrade kubernetes, whenever they adopt PyYAML 5
Test [122/128] Test that boolean flags do not appear on command line if inputBinding is empty and not null
Final process status is permanentFail
Test 122 failed: /usr/local/bin/calrissian --max-ram 8G --max-cores 4 --default-container debian:stretch-slim --outdir=/output/tmpipcpuvsd --quiet v1.0/bool-empty-inputbinding.cwl v1.0/bool-empty-inputbinding-job.json
Test that boolean flags do not appear on command line if inputBinding is empty and not null
Returned non-zero
CWL provides a suite of conformance tests to demonstrate how an implementation conforms to the spec: https://github.com/common-workflow-language/cwltool/blob/master/cwltool/schemas/CONFORMANCE_TESTS.md
I've started to explore how to run these:
python -m venv venv
source venv/bin/activate
pip install calrissian cwltest
cd cwltool/cwltool/schemas
git clone [email protected]:common-workflow-language/cwltool.git
./run_test.sh RUNNER=calrissian -n1 --verbose --EXTRA="--max-ram 16G --max-cores 8"
Above attempts to run the first test, but fails:
$ ./run_test.sh RUNNER=calrissian -n1 --verbose --EXTRA="--max-ram 16G --max-cores 8"
./run_test.sh: line 69: eval: --: invalid option
eval: usage: eval [arg ...]
--- Running conformance test v1.0 on /Users/dcl9/Code/python/calrissian-conformance/venv/bin/calrissian ---
calrissian 0.5.0 (cwltool 1.0.20181217162649)
cwltest --tool /Users/dcl9/Code/python/calrissian-conformance/venv/bin/calrissian --test=conformance_test_v1.0.yaml -n1 --verbose --basedir /Users/dcl9/Code/python/calrissian-conformance/cwltool/cwltool/schemas/v1.0 --
Test [1/128] General test of command line generation
Test 1 failed: /Users/dcl9/Code/python/calrissian-conformance/venv/bin/calrissian --outdir=/var/folders/7t/q8bb2np92p5b26pk_h5lr0lh0000gn/T/tmp3sqc2xvb --quiet v1.0/bwa-mem-tool.cwl v1.0/bwa-mem-job.json
General test of command line generation
Returned non-zero
0 tests passed, 1 failures, 0 unsupported features
1 tool tests failed
Notes:
EXTRA
option. I'm not sure if it's being used correctlyTo aid development, I created a DeploymentConfig to build this application from source and run it like a web app. Then, to try running a workflow, I oc rsh
into the pod and invoke things there.
So this issue exists to remind us of this pattern and clean it up when we have something better.
The current job specification has a command that runs run.sh in bash:
And the CWL engine runs just fine.
I attempted to change this command in the YAML to remove the intermediate shell script:
command: ["python", "-m", "calrissian.main", "--tmpdir-prefix", "/calrissian/tmptmp/", "--tmp-outdir-prefix", "/calrissian/tmpout", "--outdir", "/calrissian/output-data", "/calrissian/input-data/revsort-array.cwl", "/calrissian/input-data/revsort-array-job.json"]
But the CWL process fails:
cwltool.sandboxjs.JavascriptException: cwltool requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:slim
If I debug the failed container, node
is present and available:
I think this is available to shells but not if the container command is bash. Something to watch out for.
Detail how calrissian can be used with minishift.
Two issues that I am aware of are built image url differences and a persistent volume limitation.
When --tmp-outdir-prefix /calrissian/tmpout
is used the following error occurs:
...
Unhandled error, try again with --debug for more information:
--
| [Errno 13] Permission denied: '/calrissian/tmpoutocot8g90'
I believe this should have been /calrissian/tmpout/ocot8g90
When --tmpdir-prefix /calrissian/tmptmp
is used the following error occurs:
Unexpected exception
--
| Traceback (most recent call last):
| File "/opt/app-root/lib/python3.6/site-packages/cwltool/workflow.py", line 793, in job
| runtimeContext):
| File "/opt/app-root/lib/python3.6/site-packages/cwltool/command_line_tool.py", line 500, in job
| tempfile.mkdtemp(prefix=tmpdir_prefix) # type: ignore
| File "/opt/app-root/lib64/python3.6/tempfile.py", line 368, in mkdtemp
| _os.mkdir(file, 0o700)
| PermissionError: [Errno 13] Permission denied: '/calrissian/tmptmp9z0_79if'
| Cannot make scatter job: [Errno 13] Permission denied: '/calrissian/tmptmp9z0_79if'
| Workflow cannot make any more progress.
I believe this should have been /calrissian/tmptmp/9z0_79if
In attempts to run exomeseq-gatk4-preprocessing, the map step fails because the command is not constructed correctly.
Relevant snippet from generated spec:
containers:
- args:
- >-
bash bwa-mem-samtools.sh -R
@RG\tID:SA05051\tLB:F012354\tPL:Illumina\tPU:SA05051\tSM:SA05051 -t 20
/var/lib/cwl/stgc9814adc-1982-403b-ade3-e106d7a82960/human_g1k_v37_decoy.fasta
/var/lib/cwl/stgec2e16b0-4425-4dcc-8baa-dd9b5bef0451/SA05051-R1_val_1.fq.gz
/var/lib/cwl/stg4f3a9d6b-7759-40fb-8070-ffecf1403d45/SA05051-R2_val_2.fq.gz
> SA05051-mapped.bam
command:
- /bin/sh
- '-c'
env:
- name: HOME
value: /plpTRq
- name: TMPDIR
value: /tmp
image: 'broadinstitute/genomes-in-the-cloud:2.3.1-1512499786'
I believe that the base cwltool job would catch this with
but this feature is still outstanding in #8
Test [89/128] Test file literal as input
Got workflow error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cwltool/executors.py", line 264, in runner
job.run(runtime_context)
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 554, in run
pod = self.create_kubernetes_runtime(runtimeContext) # analogous to create_runtime()
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 429, in create_kubernetes_runtime
any_path_okay=True)
File "/usr/local/lib/python3.6/site-packages/cwltool/job.py", line 575, in add_volumes
runtime, vol, host_outdir_tgt, secret_store, tmpdir_prefix)
File "/usr/local/lib/python3.6/site-packages/cwltool/job.py", line 537, in create_file_and_add_volume
writable=writable)
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 579, in append_volume
raise NotImplementedError('append_volume')
NotImplementedError: append_volume
Workflow error, try again with --debug for more information:
append_volume
Test 89 failed: /usr/local/bin/calrissian --max-ram 8G --max-cores 4 --default-container debian:stretch-slim --outdir=/output/tmp6g8tvjsk --quiet v1.0/cat3-tool.cwl v1.0/file-literal.yml
Test file literal as input
Returned non-zero
As noted in
Line 253 in b4526af
DockerCommandLineJob.create_runtime() builds up some additional docker command flags (read-only container filesystem, custom network access, matching user, etc)
Some of these may be necessary/appropriate. Should research/triage/implement as needed
Test [119/128] Test file literal as input without Docker
Got workflow error
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/cwltool/executors.py", line 264, in runner
job.run(runtime_context)
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 554, in run
pod = self.create_kubernetes_runtime(runtimeContext) # analogous to create_runtime()
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 429, in create_kubernetes_runtime
any_path_okay=True)
File "/usr/local/lib/python3.6/site-packages/cwltool/job.py", line 575, in add_volumes
runtime, vol, host_outdir_tgt, secret_store, tmpdir_prefix)
File "/usr/local/lib/python3.6/site-packages/cwltool/job.py", line 537, in create_file_and_add_volume
writable=writable)
File "/usr/local/lib/python3.6/site-packages/calrissian/job.py", line 579, in append_volume
raise NotImplementedError('append_volume')
NotImplementedError: append_volume
Workflow error, try again with --debug for more information:
append_volume
Test 119 failed: /usr/local/bin/calrissian --max-ram 8G --max-cores 4 --default-container debian:stretch-slim --outdir=/output/tmpb66gmonw --quiet v1.0/cat3-nodocker.cwl v1.0/file-literal.yml
Test file literal as input without Docker
Returned non-zero
Calrissian creates a Kubernetes Job
for each CWL CommandLineTool job to run. The KubernetesJobBuilder.build method builds the single-container definition.
However, to determine the CWL CommandLineTool's exit code, #21 makes a change to inspect each k8s Job's first Pod, and includes some code to check that the job had 1 pod and the pod had 1 container.
After implementing this, I'm not sure the k8s Job is what we should be submitting. Jobs are controllers in k8s and can have multiple replicas or cron behaviors. Additionally, k8s will restart a Job's pod(s) on a different node if it doesn't complete once. I don't think we need that layer of management, since calrissian is already watching/managing those details.
tl;dr: can we update calrissian to submit Pods instead of Jobs? It would probably simplify/streamline the code that's here.
CWL uses a tmpout
volume to store intermediate results between each step of a workflow.
Right now we are using a ReadWriteMany volume to share data between the pods running each step.
Look into creating multiple volumes with ReadWriteOnce
instead of a single large ReadWriteMany
to improve performance.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.