The Common Workflow Language (CWL) is a specification for describing analysis
workflows and tools in a way that makes them portable and scalable across a
variety of software and hardware environments, from workstations to cluster,
cloud, and high performance computing (HPC) environments. CWL is designed to
meet the needs of data-intensive science, such as Bioinformatics, Medical
Imaging, Astronomy, Physics, and Chemistry.
CWL is developed by a multi-vendor working group consisting of
organizations and individuals aiming to enable scientists to share data
analysis workflows. The CWL project is maintained on
Github and we follow the Open-Stand.org principles for
collaborative open standards development. Legally CWL is a member project of
Software Freedom Conservancy and is formally managed by
the elected CWL leadership team, however every-day project decisions are made by the
CWL community which is open for participation by anyone.
CWL builds on technologies such as JSON-LD for data modeling and Docker for portable runtime environments.
The CWL user guide provides a gentle introduction to learning how to write CWL command line tool and workflow descriptions.
CWLの日本語での解説ドキュメント is a 15 minute introduction to the CWL project in Japanese.
For developers and advanced users, the current CWL specification (v1.0.2) provides authoritative documentation of the execution of CWL documents.
To reference the CWL standards in a scholary work, please use the following citation inclusive of the DOI:
Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić (editors), Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes, Luka Stojanovic (2016): Common Workflow Language, v1.0. Specification, Common Workflow Language working group. https://w3id.org/cwl/v1.0/ doi:10.6084/m9.figshare.3115156.v2
A collection of existing references to CWL can be found at http://www.citeulike.org/group/20713
Software | Description | CWL support | Platform support |
---|---|---|---|
cwltool | Reference implementation of CWL | Linux, OS X, Windows, local execution only | |
Arvados | Distributed computing platform for data analysis on massive data sets. Using CWL on Arvados | AWS, GCP, Azure, Slurm | |
Toil | Toil is a workflow engine entirely written in Python. | AWS, Azure, GCP, Grid Engine, HTCondor, LSF, Mesos, OpenStack, Slurm, PBS/Torque | |
Rabix Bunny | An open-source, Java-based implementation of Common Workflow Language with support for multiple drafts/versions. See Rabix.io for details. | Linux, OS X, GA4GH TES (experimental) | |
Galaxy | Web-based platform for data intensive biomedical research. | alpha | - |
cwl-tes | CWL engine backended by the GA4GH Task Execution API | Alicloud, AWS, Google, HPC, Local, Spark, TES | |
CWL-Airflow | Package to run CWL workflows in Apache-Airflow (supported by BioWardrobe Team, CCHMC) | Linux, OS X | |
REANA | RE usable ANAlyses | Yes | Kubernetes, CERN OpenStack (OpenStack Magnum) |
Cromwell | Cromwell workflow engine | Google, HTCondor, Local, LSF, PBS/Torque, SGE, Slurm, TES | |
CWLEXEC | Apache 2.0 licensed CWL executor for IBM Spectrum LSF, supported by IBM for customers with valid contracts. | IBM Spectrum LSF 10.1.0.3+ | |
Xenon | Run CWL workflows using Xenon | alpha | any Xenon backend: local, ssh, SLURM, Torque, Grid Engine |
Consonance | orchestration tool for running SeqWare workflows and CWL tools | - | AWS, OpenStack, Azure |
Apache Taverna | Domain-independent Workflow Management System | alpha | - |
AWE | Workflow and resource management system for bioinformatics data analysis. | - | |
yacle | Yet Another CWL Engine | local |
Repository | Description |
---|---|
Workflows repository | Git repository of user contributed tools and workflows. |
Dockstore tool registry | An open platform for sharing Docker-based tools described with the Common Workflow Language used by the GA4GH. |
CWLviewer | A web application to view and share Common Workflow Language workflows |
GitHub | Search for CWL documents using extension:cwl cwlVersion + <your search terms> , for example extension:cwl cwlVersion picard . |
Search for CWL documents using filetype:cwl cwlVersion + <your search terms> , for example filetype:cwl cwlVersion picard . |
Software | Description |
---|---|
Rabix Composer | Graphical CWL editor |
CWLviewer | A web application to view and share Common Workflow Language workflows |
atom-cwl | CWL editing mode for Atom |
vim-cwl | CWL editing mode for Vim |
cwl-mode | CWL editing mode for Emacs (instructions english, 日本語) |
vscode-cwl | CWL support in Visual Studio Code |
IntelliJ CWL plugin | CWL plugin for IntelliJ and other JetBrains editors |
bioSyntax | Includes CWL syntax highliting for gedit |
sublime-cwl-syntax | Sublime Text syntax highlighting for CWL |
Software | Description |
---|---|
cwldep | CWL dependency manager, for importing tool wrappers and workflows into your own project. |
cwltest | CWL testing framework, automated testing of tools and workflows written with CWL |
cwl2zshcomp | generates ZSH auto completions from CWL command line tool descriptions |
Cerise | A REST service for running CWL workflows on remote clusters |
cwl-inspector | Tool to inspect properties of tools or workflows written in CWL |
cwlprov-py | Command line tool and Python API to explore CWLProv Research Objects containing provenance of Common Workflow Language executions |
Software | Description |
---|---|
cwl-upgrader | Upgrade CWL documents from draft-3 to v1.0 |
argparse2tool | Generate CWL CommandLineTool wrappers (and/or Galaxy tool descriptions) from Python programs that use argparse. Also supports the click argument parser. |
cwl2argparse | Generate Python argparse code from CWL CommandLineTool description. |
pypi2cwl | Automatically run argparse2cwl on any package in PyPi |
acd2cwl | ACD (EMBOSS) to CWL generator |
CTD converter | Common Tool Definition (CTD) to CWL converter |
scriptcwl | Create CWL workflows by writing a simple Python script |
python-cwlgen | Generate of CWL programmatically from Python. |
cwl2nxf | Convert CWL to run on Nextflow |
cwl-to-parsl | Convert CWL to Parsl |
Beatrice | Pipeline Assembler For CWL |
Software | Description |
---|---|
cwltool | cwltool (can be imported as a Python module and extended to create custom cwl runners |
schema salad | Python module and tools for working with the CWL schema. |
cwlavro | Java classes for loading CWL documents |
CWL for R | Parse and work with CWL from R |
buchanae/cwl | CWL document parsing and processing utilities in Go. |
CWL for Go | - |
CWL for Scala | CWL object model for Scala |
cwl-proto | Reading and writing Common Workflow Language to Protocol Buffers |
CmdParser | Reading and Writing Common Workflow Language spec files from C++ applications. Includes a Command Line Parser |
Name | Details |
---|---|
Bio-compute objects | "a step towards evaluation and validation of bio-medical scientific computations", CWL and researchobject.org participants are cooperating with this effort |
European Open Science Cloud | The CWL project is signatory to the EOSC Declaration and is a core enabling standard for two of the EOSCPilot Science Demonstrations: LOFAR - Astrophysics Data and eWaterCycle & SWITCH-ON – FAIR Data for Hydrology |
GA4GH Task Execution API | a minimal common API for submitting a single job to a remote execution endpoint. Many contributions from CWL project participants. |
GA4GH Workflow Execution API | a minimal common API for submitting workflow requests to workflow execution systems in a standardized way. Many contributions from CWL project participants. |
ResearchObjects.org | "an emerging approach to the publication, and exchange of scholarly information on the Web." CWL participants and RO enthusiasts have created CWLProv, a profile for provenance research object of a CWL workflow run. |
The recommended place to ask a question about all things CWL is on Biostars. After you have read previous questions and answers you can post your question using the 'cwl' tag
If you are interested in learning more or contributing ideas or code, come chat with us on Gitter, check out #CommonWL on Twitter, join the mailing list common-workflow-language on Google Groups or fork the repository and send a pull request!
Besides the web interface for the mailing list, one can also join by sending a blank email to [email protected] and replying to the automated message.
The CWL Project is dedicated to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, age, race, or religion. We do not tolerate harassment of participants in any form. This code of conduct applies to all CWL Project spaces, including the Google Group, the Gitter chat room, the Google Hangouts chats, both online and off. Anyone who violates this code of conduct may be sanctioned or expelled from these spaces at the discretion of the leadership team.
For more details, see our Code of Conduct.
- Curoverse
- Seven Bridges Genomics
- Galaxy Project
- Apache Taverna
- Institut Pasteur
- Wellcome Trust Sanger Institute
- University of California Santa Cruz
- Harvard T.H. Chan School of Public Health
- Cincinnati Children's Hospital Medical Center
- Broad Institute
- University of Melbourne Center for Cancer Research
- Netherlands eScience Center
- Texas Advanced Computing Center Life Science Computing Group / Agave Platform
- CyVerse
- Institute for Systems Biology
- ELIXIR Europe
- BioExcel CoE
- BD2K
- EMBL Australia Bioinformatics Resource
- IBM Spectrum Computing
- DNAnexus
- CERN
(Alphabetical)
- Peter Amstutz, Curoverse Inc. / Arvados; https://orcid.org/0000-0003-3566-7705
- Robin Andeer; https://orcid.org/0000-0003-1132-5305
- Brad Chapman; https://orcid.org/0000-0002-3026-1856
- John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756
- Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670
- Roman Valls Guimerà; https://orcid.org/0000-0002-0034-9697
- Guillermo Carrasco Hernandez [email protected]
- Kenzo-Hugo Hillion; https://orcid.org/0000-0002-6517-6934
- Manabu Ishii, RIKEN; https://orcid.org/0000-0002-5843-4712
- Sinisa Ivkovic [email protected]
- Sehrish Kanwal; https://orcid.org/0000-0002-5044-4692
- Andrey Kartashov; https://orcid.org/0000-0001-9102-5681
- John Kern; https://orcid.org/0000-0001-6977-458X
- Farah Zaib Khan; https://orcid.org/0000-0002-6337-3037
- Dan Leehr; https://orcid.org/0000-0003-3221-9579
- Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009
- Maxim Mikheev [email protected]
- Michael Miller [email protected]
- Tazro Ohta, DBCLS; http://orcid.org/0000-0003-3777-5945
- Tim Pierce [email protected]
- Josh Randall; https://orcid.org/0000-0003-1540-203X
- Mark Robinson; https://orcid.org/0000-0002-8184-7507
- Janko Simonović [email protected]
- Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718
- Luka Stojanovic [email protected]
- Tomoya Tanjo, NII; https://orcid.org/0000-0002-4421-9659
- Nebojša Tijanić [email protected]
- Hiromu Ochiai; @otiai10 https://orcid.org/0000-0001-6636-856X
(Alphabetical)
- Peter Amstutz, Curoverse Inc. / Arvados; https://orcid.org/0000-0003-3566-7705
- Artem Barski, Cincinnati Children's Hospital Medical Center / University of Cincinnati College of Medicine; https://orcid.org/0000-0002-1861-5316
- John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756
- Kyle Cranmer, New York University; https://orcid.org/0000-0002-5769-7094
- Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670
- Brandi Davis Dusenbery, Seven Bridges Genomics, Inc.; https://orcid.org/0000-0001-7811-8613
- Niels Drost, Netherland eScience Center; https://orcid.org/0000-0001-9795-7981
- Geet Duggal, DNAnexus; https://orcid.org/0000-0003-3485-359X
- Rob Finn, EMBL-EBI; https://orcid.org/0000-0001-8626-2148
- Marc Fiume, DNAstack; https://orcid.org/0000-0002-9769-375X
- Jeff Gentry, Broad Institute; https://orcid.org/0000-0001-5351-8442
- Kaushik Ghose, Seven Bridges Genomics, Inc; https://orcid.org/0000-0003-2933-1260
- Carole Goble, The University of Manchester; https://orcid.org/0000-0003-1219-2137
- Oliver Hofmann, University of Melbourne / bcbio-nextgen; https://orcid.org/0000-0002-7738-1513
- Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009
- Folker Meyer, Argonne / University of Chicago; https://orcid.org/0000-0003-1112-2284
- Tom Morris, Curoverse; https://orcid.org/0000-0003-0435-7851
- Anton Nekrutenko, The Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-5987-8032
- Brian O'Connor, University of California Santa Cruz; https://orcid.org/0000-0002-7681-6415
- Tibor Simko, CERN, https://orcid.org/0000-0001-7202-5803
- Nihar Sheth, DNAnexus; https://orcid.org/0000-0003-4128-4364
- Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718
- James Taylor, Johns Hopkins University / Galaxy Project; https://orcid.org/0000-0001-5079-840X
- Nebojša Tijanić, Seven Bridges
- Ward Vandewege, Curoverse Inc. / Arvados; https://orcid.org/0000-0002-2527-6949
- Alexander Wait Zaranek, Curoverse Inc. / Arvados; https://orcid.org/0000-0002-0415-9655
(Alphabetical)
- Peter Amstutz, Curoverse Inc. / Arvados; https://orcid.org/0000-0003-3566-7705
- John Chilton, Pennsylvania State University / Galaxy Project; https://orcid.org/0000-0002-6794-0756
- Michael R. Crusoe, CWL Project Lead; https://orcid.org/0000-0002-2961-9670
- Brandi Davis Dusenbery, Seven Bridges Genomics, Inc.; https://orcid.org/0000-0001-7811-8613
- Jeff Gentry, Broad Institute; https://orcid.org/0000-0001-5351-8442
- Hervé Ménager, Institut Pasteur; https://orcid.org/0000-0002-7552-1009
- Stian Soiland-Reyes, University of Manchester; https://orcid.org/0000-0001-9842-9718