common-workflow-language/schema_salad

Semantic Annotations for Linked Avro Data

Common Workflow Language

Main website: https://www.commonwl.org

GitHub repository for www.commonwl.org: https://www.github.com/common-workflow-language/cwl-website

CWL v1.0.x: https://github.com/common-workflow-language/common-workflow-language (this repository)

CWL v1.1.x: https://github.com/common-workflow-language/cwl-v1.1/

CWL v1.2.x: https://github.com/common-workflow-language/cwl-v1.2/

Support

The Common Workflow Language (CWL) is a specification for describing analysis workflows and tools in a way that makes them portable and scalable across a
variety of software and hardware environments, from workstations to cluster, cloud, and high performance computing (HPC) environments. CWL is designed to meet the needs of data-intensive science, such as Bioinformatics, Medical Imaging, Astronomy, Physics, and Chemistry.

CWL is developed by a multi-vendor working group consisting of organizations and individuals aiming to enable scientists to share data analysis workflows. The CWL project is maintained on Github and we follow the Open-Stand.org principles for collaborative open standards development. Legally, CWL is a member project of Software Freedom Conservancy and is formally managed by the elected CWL leadership team, however every-day project decisions are made by the CWL community which is open for participation by anyone.

CWL builds on technologies such as JSON-LD for data modeling and Docker for portable runtime environments.

User Guide

The CWL user guide provides a gentle introduction to learning how to write CWL command line tool and workflow descriptions.

CWLの日本語での解説ドキュメント is a 15 minute introduction to the CWL project in Japanese.

CWL Recommended Practices

CWLの日本語での解説ドキュメント is a 15 minute introduction to the CWL project in Japanese.

A series of video lessons about CWL is available in Russian as part of the Управление вычислениями(Computation Management) free online course.

Citation

To reference the CWL project in a scholary work, please use the following citation:

Michael R. Crusoe, Sanne Abeln, Alexandru Iosup, Peter Amstutz, John Chilton, Nebojša Tijanić, Hervé Ménager, Stian Soiland-Reyes, Bogdan Gavrilović, Carole Goble, and The CWL Community. (2022): Methods Included: Standardizing Computational Reuse and Portability with the Common Workflow Language. Commun. ACM 65, 6 (June 2022), 54–63. https://doi.org/10.1145/3486897

To cite version 1.0 of the CWL standards specifically, please use the following citation inclusive of the DOI.

Peter Amstutz, Michael R. Crusoe, Nebojša Tijanić (editors), Brad Chapman, John Chilton, Michael Heuer, Andrey Kartashov, Dan Leehr, Hervé Ménager, Maya Nedeljkovich, Matt Scales, Stian Soiland-Reyes, Luka Stojanovic (2016): Common Workflow Language, v1.0. Specification, Common Workflow Language working group. https://w3id.org/cwl/v1.0/ doi:10.6084/m9.figshare.3115156.v2

A collection of existing references to CWL can be found at https://zotero.org/groups/cwl

Code of Conduct

The CWL Project is dedicated to providing a harassment-free experience for everyone, regardless of gender, gender identity and expression, sexual orientation, disability, physical appearance, body size, age, race, or religion. We do not tolerate harassment of participants in any form. This code of conduct applies to all CWL Project spaces, including the Google Group, the Gitter chat room, the Google Hangouts chats, both online and off. Anyone who violates this code of conduct may be sanctioned or expelled from these spaces at the discretion of the leadership team.

For more details, see our Code of Conduct.

For the following content:

• Support, Community and Contributing
• CWL Implementations
• Repositories of CWL Tools and Workflows
• Software for working with CWL
  • Editors and viewers
  • Utilities
  • Converters and code generators
  • Code libraries
• Projects the CWL community is participating in
• Participating Organizations
• Individual Contributors
• CWL Advisors
• CWL Leadership team

Please see https://www.commonwl.org

schema_salad's Issues

Can codegen generate CWL __version__ string

Something like __cwl_version__ or __version__ in the generated code based on what is in the schema document? Thanks!

Add support for hypothes.is to the schema_salad.makedoc for crowdsourced annotations

• Add <script defer async src="//hypothes.is/embed.js"></script> to webpages

Resolving absolute http URIs from file base path breaks on Windows

The fixes in #120 added a regression, as now on win32 the below resolution of an absolute path is wrong:

fetcher.urljoin("file:///C:/Users/fred/foo.cwl", "http://example.com/bar/soup.cwl")

returns file://example.com/bar/soup.cwl instead of the http URI as-is.

The code in question is kind-of trying to follow RFC 8089 Appendix E.2 - which highlight another edge case:

fetcher.urljoin("file:///d:/baz/soup.cwl", "/foo/soup.cwl")

should then preserve the d: drive and become file:///d:/foo/soup.cwl

schema-salad-1.12.20160610104117.tar.gz has unwanted files in it

I noticed this while testing a fresh install of Planemo where this was in the install output:

Searching for schema-salad==1.12.20160610104117
Reading https://pypi.python.org/simple/schema-salad/
Best match: schema-salad 1.12.20160610104117
Downloading https://pypi.python.org/packages/51/99/cae16cfd2e981702441011e572445e12fdd0f8f69070a8ee13d09109120b/schema-salad-1.12.20160610104117.tar.gz#md5=92635d8e1317aa161f4a35881076dc33
Processing schema-salad-1.12.20160610104117.tar.gz
Writing /tmp/easy_install-42E9pF/schema-salad-1.12.20160610104117/setup.cfg
Running schema-salad-1.12.20160610104117/setup.py -q bdist_egg --dist-dir /tmp/easy_install-42E9pF/schema-salad-1.12.20160610104117/egg-dist-tmp-Jiro6f
  File "build/bdist.macosx-10.11-intel/egg/schema_salad/ref_resolver_BACKUP_28804.py", line 222
    <<<<<<< 973dceb3aeef33920968fa257c7b04885a3a716d
     ^
SyntaxError: invalid syntax

Adding schema-salad 1.12.20160610104117 to easy-install.pth file
Installing schema-salad-tool script to /usr/local/bin

Verified with a direct download of schema-salad-1.12.20160610104117.tar.gz from PyPI:

$ curl -L -O https://pypi.python.org/packages/51/99/cae16cfd2e981702441011e572445e12fdd0f8f69070a8ee13d09109120b/schema-salad-1.12.20160610104117.tar.gz#md5=92635d8e1317aa161f4a35881076dc33
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 56403  100 56403    0     0   198k      0 --:--:-- --:--:-- --:--:--  198k
$ tar -zxvf schema-salad-1.12.20160610104117.tar.gz#md5=92635d8e1317aa161f4a35881076dc33
...
x schema-salad-1.12.20160610104117/schema_salad/ref_resolver_BACKUP_28804.py
...
x schema-salad-1.12.20160610104117/schema_salad/ref_resolver_BASE_28804.py
...
x schema-salad-1.12.20160610104117/schema_salad/ref_resolver_REMOTE_28804.py
x schema-salad-1.12.20160610104117/schema_salad/ref_resolver_LOCAL_28804.py
...

One of the four extra files, schema-salad-1.12.20160610104117/schema_salad/ref_resolver_BACKUP_28804.py file looks like a failed merge from commit 973dceb3aeef33920968fa257c7b04885a3a716d (Use scopedRef and typeDSL in metaschema), and is therefore not valid Python. e.g.

<<<<<<< 973dceb3aeef33920968fa257c7b04885a3a716d

=======
                if "refScope" in value:
                    self.scoped_ref_fields[key] = value["refScope"]
                if value.get("typeDSL"):
                    self.type_dsl_fields.add(key)
>>>>>>> Use scopedRef and typeDSL in metaschema

This is fixed in the subsequence release schema-salad-1.13.20160630144631.tar.gz (none of these extra files are present) but perhaps something in the build process can be tweaked to prevent this?

Fix Py3 on appveyor

See https://ci.appveyor.com/project/mr-c/schema-salad/build/.276-master

Drop 'required' column

It is always "true" which is confusing.

tests aren't shipped in PyPI

needs entry point for makedoc

$ schema-salad-tool --print-doc schema_salad/metaschema/metaschema.yml       
TypeError: avrold_doc() takes exactly 6 arguments (2 given)

Validation checks for duplicate ids

Validation should include a check that identifiers are not duplicated in a document.

get our typeshed merged upstream and in the next mypy release

• argparse
• avro
• mistune
• pkg_resources
• pprint
• rdflib
• requests
• ruamel
• urlparse

If they aren't present already then permission will have to be obtained

• mypy is updated to a typeshed that includes all of the above
• mypy has a release with the updated typeshed

remove dependency on avro/avro-cwl

• Run the schema_salad, cwltool unit tests and the CWL conformance tests while measuring the line coverage of the avro library.
• Remove everything that wasn't used and move the remaining code into schema_salad
• refactor and condense

If TMP or HOME not in os.environ, ref_resolver.fetch_text fails

When ref_resolver.Loader creates a session, it initializes a FileCache, passing a string in as the octal file mode. When attempting to write to the cached file, a TypeError('an integer is required') because the file mode is a string.

schema_salad/schema_salad/ref_resolver.py

Line 293 in 6ce2b9a

cache=FileCache("/tmp", ".cache", "salad"))

This is missing an os.path.join(), like the others in the function.

I encountered this running CWL's load_tool in a web application, under an apache process that does not have HOME or TMP set.

Additional info in Duke-GCB/bespin-api#128

Support scopedRef on docParent &c

pip install, avrò requirement, schema salad 1.7.20160316203940

pip install --upgrade schema-salad                           
The directory '/Users/porter/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/Users/porter/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting schema-salad
  Downloading schema-salad-1.7.20160316203940.tar.gz
Collecting requests (from schema-salad)
  Downloading requests-2.9.1-py2.py3-none-any.whl (501kB)
    100% |████████████████████████████████| 501kB 1.2MB/s 
Requirement already up-to-date: PyYAML in /Library/Python/2.7/site-packages (from schema-salad)
Collecting rdflib>=4.1.0 (from schema-salad)
  Downloading rdflib-4.2.1.tar.gz (889kB)
    100% |████████████████████████████████| 890kB 930kB/s 
Requirement already up-to-date: rdflib-jsonld>=0.3.0 in /Library/Python/2.7/site-packages/rdflib_jsonld-0.3-py2.7.egg (from schema-salad)
Collecting mistune (from schema-salad)
  Downloading mistune-0.7.2-py2.py3-none-any.whl
Requirement already up-to-date: typing in /Library/Python/2.7/site-packages/typing-3.5.0.1-py2.7.egg (from schema-salad)
Collecting avro (from schema-salad)
  Downloading avro-1.8.0.tar.gz (43kB)
    100% |████████████████████████████████| 51kB 4.3MB/s 
Requirement already up-to-date: isodate in /Library/Python/2.7/site-packages (from rdflib>=4.1.0->schema-salad)
Collecting pyparsing (from rdflib>=4.1.0->schema-salad)
  Downloading pyparsing-2.1.0-py2.py3-none-any.whl
Collecting SPARQLWrapper (from rdflib>=4.1.0->schema-salad)
  Downloading SPARQLWrapper-1.7.6.tar.gz
Requirement already up-to-date: html5lib in /Library/Python/2.7/site-packages (from rdflib>=4.1.0->schema-salad)
Collecting keepalive>=0.5 (from SPARQLWrapper->rdflib>=4.1.0->schema-salad)
  Downloading keepalive-0.5.tar.gz
Requirement already up-to-date: six in /Library/Python/2.7/site-packages (from html5lib->rdflib>=4.1.0->schema-salad)
Installing collected packages: requests, pyparsing, keepalive, SPARQLWrapper, rdflib, mistune, avro, schema-salad
  Found existing installation: requests 2.8.1
    Uninstalling requests-2.8.1:
      Successfully uninstalled requests-2.8.1
  Found existing installation: pyparsing 2.0.6
    Uninstalling pyparsing-2.0.6:
      Successfully uninstalled pyparsing-2.0.6
  Found existing installation: keepalive 0.4.1
    Uninstalling keepalive-0.4.1:
      Successfully uninstalled keepalive-0.4.1
  Running setup.py install for keepalive ... done
  Found existing installation: SPARQLWrapper 1.6.4
    Uninstalling SPARQLWrapper-1.6.4:
      Successfully uninstalled SPARQLWrapper-1.6.4
  Running setup.py install for SPARQLWrapper ... done
  Found existing installation: rdflib 4.2.0
    Uninstalling rdflib-4.2.0:
      Successfully uninstalled rdflib-4.2.0
  Running setup.py install for rdflib ... done
  Found existing installation: mistune 0.7.1
    Uninstalling mistune-0.7.1:
      Successfully uninstalled mistune-0.7.1
  Found existing installation: avro 1.7.7
    Uninstalling avro-1.7.7:
      Successfully uninstalled avro-1.7.7
  Running setup.py install for avro ... done
  Found existing installation: schema-salad 1.7.20160315232142
    Uninstalling schema-salad-1.7.20160315232142:
      Successfully uninstalled schema-salad-1.7.20160315232142
  Running setup.py install for schema-salad ... done
Successfully installed SPARQLWrapper avro keepalive mistune pyparsing rdflib requests schema-salad
Traceback (most recent call last):
  File "/usr/local/bin/pip", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/pip/__init__.py", line 213, in main
    return command.main(cmd_args)
  File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 246, in main
    pip_version_check(session)
  File "/Library/Python/2.7/site-packages/pip/utils/outdated.py", line 102, in pip_version_check
    installed_version = get_installed_version("pip")
  File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 848, in get_installed_version
    working_set = pkg_resources.WorkingSet()
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 626, in __init__
    self.add_entry(entry)
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 682, in add_entry
    for dist in find_distributions(entry, True):
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2080, in find_eggs_in_zip
    if metadata.has_metadata('PKG-INFO'):
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1610, in has_metadata
    return self.egg_info and self._has(self._fn(self.egg_info, name))
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1968, in _has
    return zip_path in self.zipinfo or zip_path in self._index()
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1848, in zipinfo
    return self._zip_manifests.load(self.loader.archive)
  File "/Library/Python/2.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1788, in load
    mtime = os.stat(path).st_mtime
OSError: [Errno 2] No such file or directory: '/Library/Python/2.7/site-packages/avro-1.7.7-py2.7.egg'
[email protected]:/Users/porter/Work/scidap/workflows>

Roundtrip (load/save) failures

Attempting to roundtrip the following workflow:

topmed-alignment.cwl.txt

Fails validation

Roundtrip code

import cwlpy.cwlmodels as cwlmodels
import ruamel.yaml as yaml
import sys

W1 = cwlmodels.load_document('topmed-alignment.cwl')
yaml.dump(cwlmodels.save(W1), sys.stdout, Dumper=yaml.RoundTripDumper)

$ python roundtrip.py > test.cwl
$ cwltool --validate test.cwl 
/Users/kghose/miniconda2/envs/kaggle/bin/cwltool 1.0.20180709065250
Resolved 'test.cwl' to 'file:///Users/kghose/Code/cwlpy/examples/003.sbg.topmed/test.cwl'
test.cwl:205:3:   checking item
test.cwl:205:3:     Field `class` contains undefined reference to `https://sevenbridges.comAWSInstanceType`
Tool definition failed validation:
test.cwl:191:1: checking field `outputs`
test.cwl:192:3:   checking object `topmed-alignment.cwl#output`
test.cwl:197:3:     Field `outputSource` references unknown identifier `file:///Users/kghose/Code/cwlpy/examples/003.sbg.topmed/topmed-alignment.cwl#output/topmed_post_align/output`, tried file:///Users/kghose/Code/cwlpy/examples/003.sbg.topmed/topmed-alignment.cwl#output/file:///Users/kghose/Code/cwlpy/examples/003.sbg.topmed/topmed-alignment.cwl#output/topmed_post_align/output, file:///Users/kghose/Code/cwlpy/examples/003.sbg.topmed/topmed-alignment.cwl#file:///Users/kghose/Code/cwlpy/examples/003.sbg.topmed/topmed-alignment.cwl#output/topmed_post_align/output

Generated CWL is:
test.cwl.txt

schema-salad-tool --codegen python CommonWorkflowLanguage.yml > ../cwlpy/cwlmodels.py
Current version: 2.7.20180701171628

cwltool --version
1.0.20180709065250

ruamel.yaml              0.13.13 

@tetron

teach schema salad about expressions

will improve docs

Python 3.x support for schema_salad

Get passing test locally (tox) and on travis CI in python 3.x environment

Checklist:

• python-avro support
• use fixers like modernize and futurize to fix basic issues
• integrate six library (majority done)
• handle string literals properly: u" " or from __future__ import unicode_literals (optional for the time being)
• mypy goals
  • update to latest version of mypy for --py2 mode
  • run in python3 mode
  • now using Text inplace of unicode in existing codebase
  • Add stub files for 3rd party libraries and stdlib to typeshed/3 and typeshed/2.7: Status
  • update Makefile
• edit .travis.yml, simplify tox.ini

skip_schemas is missing unit testing

#152

Needs a positive and a negative test

• validate.validate_ex
• validate_doc (via loader.skip_schema)

solo 'mapSubject' under `jsonldPredicate` not rendering in HTML

https://github.com/common-workflow-language/common-workflow-language/blob/46091d30e9c4fec1347a3f58b2a6056838a3da60/v1.0/Process.yml#L811

http://www.commonwl.org/v1.0/CommandLineTool.html#CommandLineTool

improve test code coverage

python-coverage report
Name                          Stmts   Miss Branch BrMiss  Cover
---------------------------------------------------------------
schema_salad/__init__             5      0      0      0   100%
schema_salad/__main__             3      3      0      0     0%
schema_salad/aslist               4      0      2      0   100%
schema_salad/flatten             17      7     10      5    56%
schema_salad/jsonld_context     108     13     60     12    85%
schema_salad/main               148     53     48     24    61%
schema_salad/makedoc            311    274    168    165     8%
schema_salad/ref_resolver       395     80    261     66    78%
schema_salad/schema             212     33    136     30    82%
schema_salad/validate           133     27    103     31    75%
---------------------------------------------------------------
TOTAL                          1336    490    788    333    61%

[WINDOWS] ref_resolver.py: DefaultFetcher.fetch_text fails to identify windows path

tests: test_errors.py, in method:

def test_errors(self):
        document_loader, avsc_names, schema_metadata, metaschema_loader = load_schema(
>           get_data(u"tests/test_schema/CommonWorkflowLanguage.yml"))

details:
the path passed to DefaultFetcher.fetch_text

• in windows: C:/Users/<path to directory>/schema_salad/schema_salad/tests/test_schema/CommonWorkflowLanguage.yml: returns unsupported scheme in url
• in linux: file:///mnt/c/Users/manu/Desktop/ubuntu projects/schema_salad/schema_salad/tests/test_schema/CommonWorkflowLanguage.yml: passes correctly

Need to modify the fetch_text function to identify windows path as well.

Generate a UML model from a schema_salad schema

Using a plain text UML format, such as:

• PlantUML and it's file format for class diagrams
• TextUML

codegen generated parser fixes needed

To keep track of #169 (comment)

• check that the class field matches
• check that there are no invalid unqualified fields #169
• store qualified metadata fields on the object in a special metadata field

--version doesn't work

usage: schema-salad-tool [-h] [--rdf-serializer RDF_SERIALIZER]
                         [--print-jsonld-context | --print-doc | --print-rdfs | --print-avro | --print-rdf | --print-pre | --print-index | --print-metadata | --version]
                         [--strict | --non-strict]
                         [--verbose | --quiet | --debug]
                         schema [document]
schema-salad-tool: error: too few arguments

despite

schema-salad-tool --help
usage: schema-salad-tool [-h] [--rdf-serializer RDF_SERIALIZER]
                         [--print-jsonld-context | --print-doc | --print-rdfs | --print-avro | --print-rdf | --print-pre | --print-index | -
-print-metadata | --version]
                         [--strict | --non-strict]
                         [--verbose | --quiet | --debug]
                         schema [document]

positional arguments:
  schema
  document

optional arguments:
  -h, --help            show this help message and exit
  --rdf-serializer RDF_SERIALIZER
                        Output RDF serialization format used by --print-rdf
                        (one of turtle (default), n3, nt, xml)
  --print-jsonld-context
                        Print JSON-LD context for schema
  --print-doc           Print HTML documentation from schema
  --print-rdfs          Print RDF schema
  --print-avro          Print Avro schema
  --print-rdf           Print corresponding RDF graph for document
  --print-pre           Print document after preprocessing
  --print-index         Print node index
  --print-metadata      Print document metadata
  --version             Print version
  --strict              Strict validation (unrecognized or out of place fields
                        are error)
  --non-strict          Lenient validation (ignore unrecognized fields)
  --verbose             Default logging
  --quiet               Only print warnings and errors.
  --debug               Print even more logging

schema-salad version 1.11.20160506154702 in pypi, missing in GitHub

@tetron What's with the release without a PR?

ImportError: No module named flatten

I've tried to run the tool and got the above error. Running with python 2.7.6, and installed in clean virtualenv.

codegen seems broken

schema-salad-tool --codegen java v1.0/CommandLineTool.yml
/home/michael/common-workflow-language/env/bin/schema-salad-tool Current version: 2.6.20171201034858
Schema `v1.0/CommandLineTool.yml` failed link checking:
v1.0/CommandLineTool.yml:139:3: checking object `https://w3id.org/cwl/cwl#EnvironmentDef`
v1.0/CommandLineTool.yml:145:3:   checking field `fields`
v1.0/CommandLineTool.yml:149:7:     checking object `https://w3id.org/cwl/cwl#EnvironmentDef/envValue`
v1.0/CommandLineTool.yml:150:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:153:3: checking object `https://w3id.org/cwl/cwl#CommandLineBinding`
v1.0/CommandLineTool.yml:155:3:   Field `extends` references unknown identifier `InputBinding`, tried https://w3id.org/cwl/cwl#InputBinding
v1.0/CommandLineTool.yml:189:3:   checking field `fields`
v1.0/CommandLineTool.yml:207:7:     checking object `https://w3id.org/cwl/cwl#CommandLineBinding/valueFrom`
v1.0/CommandLineTool.yml:208:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:238:3: checking object `https://w3id.org/cwl/cwl#CommandOutputBinding`
v1.0/CommandLineTool.yml:240:3:   Field `extends` references unknown identifier `OutputBinding`, tried https://w3id.org/cwl/cwl#OutputBinding
v1.0/CommandLineTool.yml:252:3:   checking field `fields`
v1.0/CommandLineTool.yml:253:7:     checking object `https://w3id.org/cwl/cwl#CommandOutputBinding/glob`
v1.0/CommandLineTool.yml:254:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:276:7:     checking object `https://w3id.org/cwl/cwl#CommandOutputBinding/outputEval`
v1.0/CommandLineTool.yml:277:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:291:3: checking object `https://w3id.org/cwl/cwl#CommandInputRecordField`
v1.0/CommandLineTool.yml:293:3:   Field `extends` references unknown identifier `InputRecordField`, tried https://w3id.org/cwl/cwl#InputRecordField
v1.0/CommandLineTool.yml:294:3:   checking field `specialize`
v1.0/CommandLineTool.yml:295:7:     checking item
v1.0/CommandLineTool.yml:295:7:       Field `specializeFrom` references unknown identifier `InputRecordSchema`, tried https://w3id.org/cwl/cwl#InputRecordSchema
v1.0/CommandLineTool.yml:297:7:     checking item
v1.0/CommandLineTool.yml:297:7:       Field `specializeFrom` references unknown identifier `InputEnumSchema`, tried https://w3id.org/cwl/cwl#InputEnumSchema
v1.0/CommandLineTool.yml:299:7:     checking item
v1.0/CommandLineTool.yml:299:7:       Field `specializeFrom` references unknown identifier `InputArraySchema`, tried https://w3id.org/cwl/cwl#InputArraySchema
v1.0/CommandLineTool.yml:301:7:     checking item
v1.0/CommandLineTool.yml:301:7:       Field `specializeFrom` references unknown identifier `InputBinding`, tried https://w3id.org/cwl/cwl#InputBinding
v1.0/CommandLineTool.yml:305:3: checking object `https://w3id.org/cwl/cwl#CommandInputRecordSchema`
v1.0/CommandLineTool.yml:307:3:   Field `extends` references unknown identifier `InputRecordSchema`, tried https://w3id.org/cwl/cwl#InputRecordSchema
v1.0/CommandLineTool.yml:308:3:   checking field `specialize`
v1.0/CommandLineTool.yml:309:7:     checking item
v1.0/CommandLineTool.yml:309:7:       Field `specializeFrom` references unknown identifier `InputRecordField`, tried https://w3id.org/cwl/cwl#InputRecordField
v1.0/CommandLineTool.yml:313:3: checking object `https://w3id.org/cwl/cwl#CommandInputEnumSchema`
v1.0/CommandLineTool.yml:315:3:   Field `extends` references unknown identifier `InputEnumSchema`, tried https://w3id.org/cwl/cwl#InputEnumSchema
v1.0/CommandLineTool.yml:316:3:   checking field `specialize`
v1.0/CommandLineTool.yml:317:7:     checking item
v1.0/CommandLineTool.yml:317:7:       Field `specializeFrom` references unknown identifier `InputBinding`, tried https://w3id.org/cwl/cwl#InputBinding
v1.0/CommandLineTool.yml:321:3: checking object `https://w3id.org/cwl/cwl#CommandInputArraySchema`
v1.0/CommandLineTool.yml:323:3:   Field `extends` references unknown identifier `InputArraySchema`, tried https://w3id.org/cwl/cwl#InputArraySchema
v1.0/CommandLineTool.yml:324:3:   checking field `specialize`
v1.0/CommandLineTool.yml:325:7:     checking item
v1.0/CommandLineTool.yml:325:7:       Field `specializeFrom` references unknown identifier `InputRecordSchema`, tried https://w3id.org/cwl/cwl#InputRecordSchema
v1.0/CommandLineTool.yml:327:7:     checking item
v1.0/CommandLineTool.yml:327:7:       Field `specializeFrom` references unknown identifier `InputEnumSchema`, tried https://w3id.org/cwl/cwl#InputEnumSchema
v1.0/CommandLineTool.yml:329:7:     checking item
v1.0/CommandLineTool.yml:329:7:       Field `specializeFrom` references unknown identifier `InputArraySchema`, tried https://w3id.org/cwl/cwl#InputArraySchema
v1.0/CommandLineTool.yml:331:7:     checking item
v1.0/CommandLineTool.yml:331:7:       Field `specializeFrom` references unknown identifier `InputBinding`, tried https://w3id.org/cwl/cwl#InputBinding
v1.0/CommandLineTool.yml:335:3: checking object `https://w3id.org/cwl/cwl#CommandOutputRecordField`
v1.0/CommandLineTool.yml:337:3:   Field `extends` references unknown identifier `OutputRecordField`, tried https://w3id.org/cwl/cwl#OutputRecordField
v1.0/CommandLineTool.yml:338:3:   checking field `specialize`
v1.0/CommandLineTool.yml:339:7:     checking item
v1.0/CommandLineTool.yml:339:7:       Field `specializeFrom` references unknown identifier `OutputRecordSchema`, tried https://w3id.org/cwl/cwl#OutputRecordSchema
v1.0/CommandLineTool.yml:341:7:     checking item
v1.0/CommandLineTool.yml:341:7:       Field `specializeFrom` references unknown identifier `OutputEnumSchema`, tried https://w3id.org/cwl/cwl#OutputEnumSchema
v1.0/CommandLineTool.yml:343:7:     checking item
v1.0/CommandLineTool.yml:343:7:       Field `specializeFrom` references unknown identifier `OutputArraySchema`, tried https://w3id.org/cwl/cwl#OutputArraySchema
v1.0/CommandLineTool.yml:345:7:     checking item
v1.0/CommandLineTool.yml:345:7:       Field `specializeFrom` references unknown identifier `OutputBinding`, tried https://w3id.org/cwl/cwl#OutputBinding
v1.0/CommandLineTool.yml:349:3: checking object `https://w3id.org/cwl/cwl#CommandOutputRecordSchema`
v1.0/CommandLineTool.yml:351:3:   Field `extends` references unknown identifier `OutputRecordSchema`, tried https://w3id.org/cwl/cwl#OutputRecordSchema
v1.0/CommandLineTool.yml:352:3:   checking field `specialize`
v1.0/CommandLineTool.yml:353:7:     checking item
v1.0/CommandLineTool.yml:353:7:       Field `specializeFrom` references unknown identifier `OutputRecordField`, tried https://w3id.org/cwl/cwl#OutputRecordField
v1.0/CommandLineTool.yml:357:3: checking object `https://w3id.org/cwl/cwl#CommandOutputEnumSchema`
v1.0/CommandLineTool.yml:359:3:   Field `extends` references unknown identifier `OutputEnumSchema`, tried https://w3id.org/cwl/cwl#OutputEnumSchema
v1.0/CommandLineTool.yml:360:3:   checking field `specialize`
v1.0/CommandLineTool.yml:361:7:     checking item
v1.0/CommandLineTool.yml:361:7:       Field `specializeFrom` references unknown identifier `OutputRecordSchema`, tried https://w3id.org/cwl/cwl#OutputRecordSchema
v1.0/CommandLineTool.yml:363:7:     checking item
v1.0/CommandLineTool.yml:363:7:       Field `specializeFrom` references unknown identifier `OutputEnumSchema`, tried https://w3id.org/cwl/cwl#OutputEnumSchema
v1.0/CommandLineTool.yml:365:7:     checking item
v1.0/CommandLineTool.yml:365:7:       Field `specializeFrom` references unknown identifier `OutputArraySchema`, tried https://w3id.org/cwl/cwl#OutputArraySchema
v1.0/CommandLineTool.yml:367:7:     checking item
v1.0/CommandLineTool.yml:367:7:       Field `specializeFrom` references unknown identifier `OutputBinding`, tried https://w3id.org/cwl/cwl#OutputBinding
v1.0/CommandLineTool.yml:371:3: checking object `https://w3id.org/cwl/cwl#CommandOutputArraySchema`
v1.0/CommandLineTool.yml:373:3:   Field `extends` references unknown identifier `OutputArraySchema`, tried https://w3id.org/cwl/cwl#OutputArraySchema
v1.0/CommandLineTool.yml:374:3:   checking field `specialize`
v1.0/CommandLineTool.yml:375:7:     checking item
v1.0/CommandLineTool.yml:375:7:       Field `specializeFrom` references unknown identifier `OutputRecordSchema`, tried https://w3id.org/cwl/cwl#OutputRecordSchema
v1.0/CommandLineTool.yml:377:7:     checking item
v1.0/CommandLineTool.yml:377:7:       Field `specializeFrom` references unknown identifier `OutputEnumSchema`, tried https://w3id.org/cwl/cwl#OutputEnumSchema
v1.0/CommandLineTool.yml:379:7:     checking item
v1.0/CommandLineTool.yml:379:7:       Field `specializeFrom` references unknown identifier `OutputArraySchema`, tried https://w3id.org/cwl/cwl#OutputArraySchema
v1.0/CommandLineTool.yml:381:7:     checking item
v1.0/CommandLineTool.yml:381:7:       Field `specializeFrom` references unknown identifier `OutputBinding`, tried https://w3id.org/cwl/cwl#OutputBinding
v1.0/CommandLineTool.yml:385:3: checking object `https://w3id.org/cwl/cwl#CommandInputParameter`
v1.0/CommandLineTool.yml:387:3:   Field `extends` references unknown identifier `InputParameter`, tried https://w3id.org/cwl/cwl#InputParameter
v1.0/CommandLineTool.yml:389:3:   checking field `specialize`
v1.0/CommandLineTool.yml:390:7:     checking item
v1.0/CommandLineTool.yml:390:7:       Field `specializeFrom` references unknown identifier `InputRecordSchema`, tried https://w3id.org/cwl/cwl#InputRecordSchema
v1.0/CommandLineTool.yml:392:7:     checking item
v1.0/CommandLineTool.yml:392:7:       Field `specializeFrom` references unknown identifier `InputEnumSchema`, tried https://w3id.org/cwl/cwl#InputEnumSchema
v1.0/CommandLineTool.yml:394:7:     checking item
v1.0/CommandLineTool.yml:394:7:       Field `specializeFrom` references unknown identifier `InputArraySchema`, tried https://w3id.org/cwl/cwl#InputArraySchema
v1.0/CommandLineTool.yml:396:7:     checking item
v1.0/CommandLineTool.yml:396:7:       Field `specializeFrom` references unknown identifier `InputBinding`, tried https://w3id.org/cwl/cwl#InputBinding
v1.0/CommandLineTool.yml:399:3: checking object `https://w3id.org/cwl/cwl#CommandOutputParameter`
v1.0/CommandLineTool.yml:401:3:   Field `extends` references unknown identifier `OutputParameter`, tried https://w3id.org/cwl/cwl#OutputParameter
v1.0/CommandLineTool.yml:403:3:   checking field `specialize`
v1.0/CommandLineTool.yml:404:7:     checking item
v1.0/CommandLineTool.yml:404:7:       Field `specializeFrom` references unknown identifier `OutputBinding`, tried https://w3id.org/cwl/cwl#OutputBinding
v1.0/CommandLineTool.yml:406:3:   checking field `fields`
v1.0/CommandLineTool.yml:407:7:     checking object `https://w3id.org/cwl/cwl#CommandOutputParameter/type`
v1.0/CommandLineTool.yml:408:7:       Field `type` references unknown identifier `CWLType`, tried https://w3id.org/cwl/cwl#CWLType
v1.0/CommandLineTool.yml:418:11:       Field `items` references unknown identifier `CWLType`, tried https://w3id.org/cwl/cwl#CWLType
v1.0/CommandLineTool.yml:408:7:       checking field `type`
:417:11:         checking item
v1.0/CommandLineTool.yml:418:11:           Field `items` references unknown identifier `CWLType`, tried https://w3id.org/cwl/cwl#CWLType
v1.0/CommandLineTool.yml:528:3: checking object `https://w3id.org/cwl/cwl#CommandLineTool`
v1.0/CommandLineTool.yml:530:3:   Field `extends` references unknown identifier `Process`, tried https://w3id.org/cwl/cwl#Process
v1.0/CommandLineTool.yml:532:3:   checking field `specialize`
v1.0/CommandLineTool.yml:533:7:     checking item
v1.0/CommandLineTool.yml:533:7:       Field `specializeFrom` references unknown identifier `InputParameter`, tried https://w3id.org/cwl/cwl#InputParameter
v1.0/CommandLineTool.yml:535:7:     checking item
v1.0/CommandLineTool.yml:535:7:       Field `specializeFrom` references unknown identifier `OutputParameter`, tried https://w3id.org/cwl/cwl#OutputParameter
v1.0/CommandLineTool.yml:540:3:   checking field `fields`
v1.0/CommandLineTool.yml:569:7:     checking object `https://w3id.org/cwl/cwl#CommandLineTool/arguments`
v1.0/CommandLineTool.yml:575:11:       Field `items` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:572:7:       checking field `type`
:574:11:         checking item
v1.0/CommandLineTool.yml:575:11:           Field `items` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:579:7:     checking object `https://w3id.org/cwl/cwl#CommandLineTool/stdin`
v1.0/CommandLineTool.yml:580:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:584:7:     checking object `https://w3id.org/cwl/cwl#CommandLineTool/stderr`
v1.0/CommandLineTool.yml:585:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:597:7:     checking object `https://w3id.org/cwl/cwl#CommandLineTool/stdout`
v1.0/CommandLineTool.yml:598:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:630:3: checking object `https://w3id.org/cwl/cwl#DockerRequirement`
v1.0/CommandLineTool.yml:632:3:   Field `extends` references unknown identifier `ProcessRequirement`, tried https://w3id.org/cwl/cwl#ProcessRequirement
v1.0/CommandLineTool.yml:699:3: checking object `https://w3id.org/cwl/cwl#SoftwareRequirement`
v1.0/CommandLineTool.yml:701:3:   Field `extends` references unknown identifier `ProcessRequirement`, tried https://w3id.org/cwl/cwl#ProcessRequirement
v1.0/CommandLineTool.yml:778:3: checking object `https://w3id.org/cwl/cwl#Dirent`
v1.0/CommandLineTool.yml:785:3:   checking field `fields`
v1.0/CommandLineTool.yml:786:7:     checking object `https://w3id.org/cwl/cwl#Dirent/entryname`
v1.0/CommandLineTool.yml:787:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:794:7:     checking object `https://w3id.org/cwl/cwl#Dirent/entry`
v1.0/CommandLineTool.yml:795:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:826:3: checking object `https://w3id.org/cwl/cwl#InitialWorkDirRequirement`
v1.0/CommandLineTool.yml:828:3:   Field `extends` references unknown identifier `ProcessRequirement`, tried https://w3id.org/cwl/cwl#ProcessRequirement
v1.0/CommandLineTool.yml:833:3:   checking field `fields`
v1.0/CommandLineTool.yml:840:7:     checking object `https://w3id.org/cwl/cwl#InitialWorkDirRequirement/listing`
v1.0/CommandLineTool.yml:843:11:       Field `items` references unknown identifier `File`, tried https://w3id.org/cwl/cwl#File
v1.0/CommandLineTool.yml:843:11:       Field `items` references unknown identifier `Directory`, tried https://w3id.org/cwl/cwl#Directory
v1.0/CommandLineTool.yml:843:11:       Field `items` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:841:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:841:7:       checking field `type`
:842:11:         checking item
v1.0/CommandLineTool.yml:843:11:           Field `items` references unknown identifier `File`, tried https://w3id.org/cwl/cwl#File
v1.0/CommandLineTool.yml:843:11:           Field `items` references unknown identifier `Directory`, tried https://w3id.org/cwl/cwl#Directory
v1.0/CommandLineTool.yml:843:11:           Field `items` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:863:3: checking object `https://w3id.org/cwl/cwl#EnvVarRequirement`
v1.0/CommandLineTool.yml:865:3:   Field `extends` references unknown identifier `ProcessRequirement`, tried https://w3id.org/cwl/cwl#ProcessRequirement
v1.0/CommandLineTool.yml:884:3: checking object `https://w3id.org/cwl/cwl#ShellCommandRequirement`
v1.0/CommandLineTool.yml:886:3:   Field `extends` references unknown identifier `ProcessRequirement`, tried https://w3id.org/cwl/cwl#ProcessRequirement
v1.0/CommandLineTool.yml:904:3: checking object `https://w3id.org/cwl/cwl#ResourceRequirement`
v1.0/CommandLineTool.yml:906:3:   Field `extends` references unknown identifier `ProcessRequirement`, tried https://w3id.org/cwl/cwl#ProcessRequirement
v1.0/CommandLineTool.yml:929:3:   checking field `fields`
v1.0/CommandLineTool.yml:936:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/coresMin`
v1.0/CommandLineTool.yml:937:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:940:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/coresMax`
v1.0/CommandLineTool.yml:941:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:944:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/ramMin`
v1.0/CommandLineTool.yml:945:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:948:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/ramMax`
v1.0/CommandLineTool.yml:949:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:952:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/tmpdirMin`
v1.0/CommandLineTool.yml:953:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:956:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/tmpdirMax`
v1.0/CommandLineTool.yml:957:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:960:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/outdirMin`
v1.0/CommandLineTool.yml:961:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression
v1.0/CommandLineTool.yml:964:7:     checking object `https://w3id.org/cwl/cwl#ResourceRequirement/outdirMax`
v1.0/CommandLineTool.yml:965:7:       Field `type` references unknown identifier `Expression`, tried https://w3id.org/cwl/cwl#Expression

Remove hook set by past.autotranslate

In init.py the past.autotranslate function is used. This function hooks the Py2Fixer loader to sys.meta_path. This is an unwanted side effect and can result in errors when importing schema_salad in python 3 code (see this issue)

I think the hooks need to be removed at some point:

from past.translation import remove_hooks
remove_hooks()

CI failed: Inconsistent definition from schema_salad

Background

Travis CI is failing everyday. That is significant problem as an OSS.

.cwl files in CWL repository are invalid as being validated by schema_salad. This is because schema_salad checks the duplicated keys of dictionary with granting whole cwl as one dictionary.

Obviously CWL requires the same key names to make references between some inputs.

It means schema_salad, supposed to be a validator for CWL, doesn't support CWL.

Suggestion

As an OSS, the health check of software is, per se, a guideline to contribute. I, as a programmer, want to suggest/pull-request following process to make this OSS healthy.

1. Temporarily allow duplicated keys in schema_salad to make the status green
2. Keep green developing CWL continuously
3. In parallel make more specific validation in schema_salad

Include License.txt in pypi source distribution

Would it be possible to include the license file in the source tar.gz on pypi? That would help packaging it at conda-forge:

Currently the archive on pypi contains:

-rw-r--r--@  1 christian  staff   397B 16 Nov 03:44 MANIFEST.in
-rw-r--r--@  1 christian  staff   6.9K 16 Nov 03:44 Makefile
-rw-r--r--@  1 christian  staff   6.8K 16 Nov 03:44 PKG-INFO
-rw-r--r--@  1 christian  staff   4.9K 16 Nov 03:44 README.rst
-rw-r--r--@  1 christian  staff   820B 16 Nov 03:44 gittaggers.py
drwxr-xr-x@ 20 christian  staff   640B 16 Nov 03:44 schema_salad
drwxr-xr-x@  9 christian  staff   288B 16 Nov 03:44 schema_salad.egg-info
-rw-r--r--@  1 christian  staff   272B 16 Nov 03:44 setup.cfg
-rwxr-xr-x@  1 christian  staff   2.8K 16 Nov 03:44 setup.py

study lockfile package deprecation situation

psf/cachecontrol#109

Use the DictCache or other custom backend?

http://cachecontrol.readthedocs.io/en/latest/storage.html#dictcache

Are there other HTTP caching packages that are good?

document /tmp fix for Windows+Py3

• File issue with futurize
• Send PR correcting their code
• Explain that this is a temporary workaround, include links to the issue filed with futurize near https://github.com/common-workflow-language/schema_salad/pull/125/files#diff-b56d3fcf71d64c4538a319d61012ca04R18 #127

A codegen-ed parser accidentally omit a property for `DockerRequirement`

I reproduced this issue with schema_salad 2.6.20180214144209 and Python 3.6.2.
I used the following CWL file (echo.cwl) for parsing.

requirements:
  - class: DockerRequirement
    dockerPull: docker/whalesay
class: CommandLineTool
cwlVersion: v1.0
baseCommand: cowsay
inputs:
  - id: input
    type: string
    inputBinding: {}
outputs:
  output:
    type: stdout
stdout: output

How to reproduce

$ git clone https://github.com/common-workflow-language/common-workflow-language.git
$ schema-salad-tool --codegen python common-workflow-language/v1.0/CommonWorkflowLanguage.yml > cwl_parser.py
$ python
>>> import cwl_parser
>>> cwl = cwl_parser.load_document('echo.cwl')
>>> cwl.requirements[0].dockerPull

Expected result

I want to get the following output:

'docker/whalesay'

Actual result

I got the following message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'InlineJavascriptRequirement' object has no attribute 'dockerPull'

Duplicate ID check fails with the same name for input and output

When checking for duplicate IDs starting from version 2.5.20170327140858, it's not possible to set input and output with the same name. Both input and output got the same identifier:
file:///blah/blah/blah/task.cwl#some_name

Testing example

cwlVersion: v1.0
class: CommandLineTool
hints:
  DockerRequirement:
    dockerPull: ubuntu
baseCommand: [touch]

inputs:
  some_name:
    type: string
    inputBinding:
      position: 1
    default: hello.txt

outputs:
  some_name:
    type: File
    outputBinding:
      glob: "hello.txt"

Detailed log

         
/usr/local/bin/cwltool 1.0.20170329142446
Resolved 'task.cwl' to 'file:///Users/kot4or/workspaces/cwl_ws/cwl_sandbox/schema_salad_dup_names/task.cwl'
Tool definition failed validation:
task.cwl:15:1: checking field `outputs`
task.cwl:16:3:   checking object `task.cwl#some_name`
task.cwl:9:3:     object id `task.cwl#some_name` previously defined
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/cwltool/main.py", line 686, in main
    fetcher_constructor=fetcher_constructor)
  File "/usr/local/lib/python2.7/site-packages/cwltool/load_tool.py", line 186, in validate_document
    processobj, metadata = document_loader.resolve_all(workflowobj, fileuri)
  File "/usr/local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 828, in resolve_all
    self.validate_links(document, u"", all_doc_ids)
  File "/usr/local/lib/python2.7/site-packages/schema_salad/ref_resolver.py", line 994, in validate_links
    raise errors[0]
ValidationException: task.cwl:15:1: checking field `outputs`
task.cwl:16:3:   checking object `task.cwl#some_name`
task.cwl:9:3:     object id `task.cwl#some_name` previously defined
         
      

Unbound variable in schema.py

What is r variable supposed to be in these few lines: https://github.com/common-workflow-language/schema_salad/blob/master/schema_salad/schema.py#L309 ?

Please tag releases

add automated test for breaking the API

to avoid situations like common-workflow-language/cwltool#987 (comment) again

Add as another task in tox.ini

• 1. Check the diff, did we change/increment the prefix in setup.py & Makefile? (also check that they match)
• 2. If not, pip install cwltool and run the cwltool tests using this version of schema_salad
• 3. Repeat with the entire suite of CWL conformance tests using the latest release of cwltool (as above)

schema-salad-tool doesn't allow for embedded subworkflows

Using a checkout of https://github.com/common-workflow-language/common-workflow-language/

$ schema-salad-tool v1.0/CommonWorkflowLanguage.yml v1.0/v1.0/count-lines10-wf.cwl
/home/michael/common-workflow-language/env/bin/schema-salad-tool Current version: 2.6.20171201034858
Document `v1.0/v1.0/count-lines10-wf.cwl` failed validation:
v1.0/v1.0/count-lines10-wf.cwl:10:1: checking field `steps`
v1.0/v1.0/count-lines10-wf.cwl:11:3:   checking object `v1.0/v1.0/count-lines10-wf.cwl#step1`
v1.0/v1.0/count-lines10-wf.cwl:12:5:     checking field `in`
v1.0/v1.0/count-lines10-wf.cwl:12:10:       checking object
                                            `v1.0/v1.0/count-lines10-wf.cwl#step1/file1`
v1.0/v1.0/count-lines10-wf.cwl:17:9:         object id
                                             `v1.0/v1.0/count-lines10-wf.cwl#step1/file1`
                                             previously defined

The contents of v1.0/v1.0/count-lines10-wf.cwl

#!/usr/bin/env cwl-runner
class: Workflow
cwlVersion: v1.0
inputs:
  file1: File
outputs:
  count_output: {type: int, outputSource: step1/count_output}
requirements:
  SubworkflowFeatureRequirement: {}
steps:
  step1:
    in: {file1: file1}
    out: [count_output]
    run:
      class: Workflow
      inputs:
        file1: File
      outputs:
        count_output: {type: int, outputSource: step2/output}
      steps:
        step1: {run: wc-tool.cwl, in: {file1: file1}, out: [output]}
        step2: {run: parseInt-tool.cwl, in: {file1: step1/output}, out: [output]}

save() on workflow throws error

Tools can be loaded and saved without any problem

Workflows load fine, error out during save

Archive.zip

Thanks!

Consider using some other system for doc generation

Instead of making our own :-)

Perhaps Sphinx? Or produce Docbook XML

Unable to compile schema using avro-tools

Hi,

This may be a followup to common-workflow-language/common-workflow-language#69 unless I'm mixing something up.

I'm using the current version of common-workflow-language (although the two tags don't seem to fare much better).

$ git status
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean
$ git log | head
commit 791f4ef1dea83aed7827b679e179c634db7c0095
Author: Peter Amstutz <[email protected]>
Date:   Mon Nov 30 14:23:50 2015 -0500

    No longer assume "#main" as entry point if there is more than one valid entry
    point in a file.

When I use the schema_salad project to generate an avro schema, that succeeds.

$ python -mschema_salad --print-avro ~/common-workflow-language/draft-3/cwl-avro.yml > cwl.avsc
/home/dyuen/schema_salad/schema_salad/__main__.py 1.0.6

However, when I attempt to compile the schema, the following happens:

$ java -jar avro-tools-1.7.7.jar compile schema cwl.avsc cwl
Input files to compile:
  cwl.avsc
Exception in thread "main" org.apache.avro.SchemaParseException: Illegal character in: draft-3.dev1
        at org.apache.avro.Schema.validateName(Schema.java:1083)
        at org.apache.avro.Schema.access$200(Schema.java:79)
        at org.apache.avro.Schema$EnumSchema.<init>(Schema.java:684)
        at org.apache.avro.Schema.parse(Schema.java:1234)
        at org.apache.avro.Schema.parse(Schema.java:1272)
        at org.apache.avro.Schema$Parser.parse(Schema.java:965)
        at org.apache.avro.Schema$Parser.parse(Schema.java:932)
        at org.apache.avro.tool.SpecificCompilerTool.run(SpecificCompilerTool.java:73)
        at org.apache.avro.tool.Main.run(Main.java:84)
        at org.apache.avro.tool.Main.main(Main.java:73)

Add a possibility to specify default values for fields in schema salad

It would be interesting to be able to specify a default value for fields in schema_salad based specs. Currently they are mentioned in the documentation field, but this is not really machine readable (e.g. https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/CommandLineTool.yml#L172). Having these would help a lot to generate SDKs ;)
cc @anton-khodak @khillion

Spaces etc in enums not escaped in RDF

As pointed out in common-workflow-language/cwltool#444 using enum symbols with spaces will cause problems with generating RDF, as the spaces are not escaped.

There are also problems with generating the JSON-LD Context if the string after the space is repeated, as it tries to insert for instance table multiple into @context:

Exception: Predicate collision on table, 'file:///c:/users/stain/src/schema_salad/schema_salad/tests/../tests/test_schema/symbols-with-spaces.yml#table_type/OTU table' != 'file:///c:/users/stain/src/schema_salad/schema_salad/tests/../tests/test_schema/symbols-with-spaces.yml#table_type/Pathway table

It is currently not defined anywhere what characters are allowed in valid schema salad identifiers - meaning anything is currently allowed, including newline, " and anything that can be expressed in YAML escapes.

A real fix is to restrict the characters of all short-hand identifiers, e.g. to be valid IRI fragment identifiers.

Some workflows already started using this, like in https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/blob/e14f5cd5e240ecb2563d138e9aa112abcdae3295/tools/biom-convert-table.yaml where constants like Pathway table (with space) is passed in as argument.

A quick fix, as in 8514cb7, will seem sufficient, but does not add required 'unquoting'. To be able to unquote we need to both have a predictable quote pattern and know all the places that needs the "pure" values. For instance the above commit breaks that workflow, as it changes the constant to Pathway_table which the command line tool would not understand.

See the https://github.com/common-workflow-language/schema_salad/tree/escape-enums branch for my naive approach (do not merge!).

Should not be allowed to resolve file: URIs from http/http

Resolving a file URI from a http/https base URI is a security concern (and reproducibility concern!)

fetcher.urljoin("http://example.com/bar/soup.cwl", "file:///C:/Users/fred/foo.cwl")

I think the above should always throw ValueError if the base URI is not in scheme file:.

In addition, considering #129 / #110 on Windows resolving C:/something should also fail if the base is not a file:

WorkflowOutputParameter should require 'type' parameter

We are allowed to create WorkflowOutputParameter without specifying type, but this will fail on validation.

Bump to v1.1-dev1

Update documentation to reflect the current master is v1.1-dev1 of metaschema, add changelog from 6ce2b9a

subclassed Classes need a better description

common-workflow-language/common-workflow-language#220

[Question] What is the license for a parser generated by `--codegen`?

I am curious about the license for a parser generated by --codegen option.
Is it licensed under Apache-2.0 (i.e. same as that for schema_salad) or others?

Trying to install on Mac OS X schema-salad-1.7.20160315232142

Collecting schema-salad
  Downloading schema-salad-1.7.20160315232142.tar.gz
    Complete output from command python setup.py egg_info:
    error in schema-salad setup command: Invalid environment marker: python_version>="2.7"

Roadmap to MS Windows support for schema_salad

• Create a separate branch: windows and start adding commits to it to get all passing tests on MS Windows.

• Setup appveyor CI and add .appveyor.yml file to the windows branch (or any other CI which provides support for Windows builds)

• Open "standing" / work-in-progress pull request

• Currently allow those test only which do not involve any unix tool requirement.

• Figure out a way to make failing tests more verbose and passing one as usual. Look into py.test args.

• Rebase and add commits from master into windows branch periodically to make sure branch doesn't go stale.

• When all the tests pass under MS Windows, add Appveyor to the list of tests that must pass before a PR is mergeable.

• Periodically merge into master so that there isn't too much divergence

• Only after schema_salad is ported to Windows, should we proceed with MS Windows support for cwltool.