elyra-ai/pipeline-schemas

pipeline-schemas

Location for the common pipeline JSON schemata and examples

Contents

This repository contains the JSON schema files pertaining to pipeline job flow authoring and execution:

The 'common-pipeline' directory

Common schemas related to pipeline definition and execution:

• Operators: Contains the base operator, uihints, and condition schemas and examples
• Pipeline-flow: Common pipeline-flow and pipeline-flow-ui JSON schemas
• Dataset-metadata: JSON dataset metadata definition

The 'common-canvas' directory

Schema and example files for driving the Common Canvas and Property Editor tooling

• Parameter-defs: Common Properties parameter editing schema and examples
• Form: Common Properties low-level form JSON specification
• Palette: Canvas palette JSON definition
• Diagram: Older (e.g. pre-pipeline-flow) internal canvas diagram specification

pipeline-schemas's Issues

Add default_value condition definition in schema

Here is a potential example -

{
  "default_value": {
    "parameter_ref": "updated_param",
    "value": "New default value",
    "evaluate": {
      "condition": {
        "parameter_ref": "some_param",
        "op": "isNotEmpty"
      }
    }
  }
}

Rename master to main

Add a new config option "resizable" in parameter_definition

Common-properties now support adjustable column headers in table - elyra-ai/canvas#842
Add a new config option "resizable" in uiHints parameter_definition.

Change newly added link_type for links to type_attr

After discussing with consumers the link_type field for links would be better named type_attr to avoid naming clashes with the type attribute for links which is visible through the Elyra canvas API.

Add https:// to enum list for pipeline-flow-v3-schema.json

Hey team, we need to change the url reference to our pipeline schema for orchestration flows to use https to fix a scan issue. The hardcoded http is getting picked up by a vulnerability scanner.
We tried changing the http to https but it seems to be failing some validator in common-canvas.js. Can we update the validator to support https as an enum?

Add a new config option for action buttons kind and size

Elyra-canvas issue - elyra-ai/canvas#1040
Added a new config option in uiHints action_info -

"button": {
          "kind": "secondary",
          "size": "lg"
 }

Tag builds that are published to npm

Done.

Setup TravisCI builds

Change comment's minimum width and height

pipeline-flow-ui-v3-schema.json has minimum width and height as 10. It should be changed to 0 as pipeline flow fails when comment's height/width is less than 10.
fyi @matthoward366

readonly controls should be able to display an icon in addition to text

Schema updates for elyra-ai/canvas#418

V3 parameter schema has wrong required property

required properties for param_property_def.properties should be name and type.
FYI @matthoward366

Add http URL to json_schema enum in pipeline flow schema

Add tearsheet support in schema

Add tearsheet support in schema

Enhance schema files to prevent schema validation issues

@tomlyn @matthoward366 @vlad-bunescu

This is a request to alter some of the pipeline-schema files to prevent schema validation errors from icorrectly occurring:

1, pipeline-flow-v3-schema.json needs to rewrite the app_data
app_data for overall pipeline, pipeline_def, execution_node_def, supernode_def, binding_entry_node_def, binding_exit_node_def, model_node_def, port_def, bound_port_def, link_def, and runtime_def uses

And here is the definition of app_data

And each of the definition in the pipeline-flow-ui-v3-schema.json allows additional properties.

This app_data specification makes it impossible for a validation tool to find what is the corresponding schema to check in the pipeline-flow-ui-v3-schema.json for a certain entity. It ends with nothing is validated for the schema defined in the pipeline-flow-ui-v3-schema.json.

2, pipeline-flow-ui-v3-schema.json
a, "required": [] in the definitions.runtime_info_def is not acceptable by Java Json Tool.
The validate requires there must be one element in the array.
As it is empty array, it is no needed to specified.

b, definitions.node_info_def.properties.messages.descriptions should be
definitions.node_info_def.properties.messages.description

3, pipeline-connection-v3-schema.json
app_data points back to pipeline-flow-v3-schema.json app_data. However, none of app_data definition in the pipeline-flow-v3-schema.json is for connection app_data.

4, datarecord-metadata-v3-schema.json
"required": [] in the definitions.runtime_info_def is not acceptable.
The validate requires there must be one element in the array.
As it is empty array, it is no needed to specified.

Binding nodes should have either required outputs or inputs in pipeline-flow-v3-schema.json

The pipeline flow schema says that the each node must be “oneOf”:
“$ref”: “#/definitions/execution_node_def”
“$ref”: “#/definitions/supernode_def”
“$ref”: “#/definitions/binding_entry_node_def”
“$ref”: “#/definitions/binding_exit_node_def”
“$ref”: “#/definitions/model_node_def”
And the definition of “oneOf” is “the given data must be valid against exactly one of the given subschemas.”

So if we have a node which only has an “id” property and a “type” property set to “binding” the node will comply with both these definitions:
“$ref”: “#/definitions/binding_entry_node_def”
“$ref”: “#/definitions/binding_exit_node_def”
because they both have “id” and “type” as required properties and for both the type property value is defined as “binding”.

So therefore such a node would not be valid against exactly one of the given definitions — and so we get schema validation errors.

To make things explicit, for “binding_entry_node_def” we should add “outputs” to the required properties list and for “binding_exit_node_def” we should add “inputs” to the required properties list since this is what is validated anyway.

Update uihints for adding buttons to tables in properties

schema changes required for elyra-ai/canvas#901

Add a new config option "tooltip_direction" for action images

Action image tooltips are now configurable - elyra-ai/canvas#898
Add a new configuration option "tooltip_direction" in uiHints.

Link ID need for link object

For API methods to retrieve and update individual links with new attributes etc the links need to be consistently referred to by the same ID. Presently, because link objects do not have IDs in the schema, internal link IDs get generated each time the piplelineFlow is loaded. The link object should have an optional id to allow any generated link IDs to be written into output pipelineFlows and also to be able to handle any IDs that a host application might want to provide.

Update uihints to support custom value in dropdowns

Update the uihints to add a new field for elyra-ai/canvas#427

Add type to complex_type_definition so parameters within the complex type can be stored as arrays or objects

Update number_generator description

Number generator link is removed from control label. Added a button next to numeric controls -

Update description to reflect these changes.

Properties panels and controls should have ability to specify a class name

elyra-ai/canvas#431

Comments should not be restricted to positive coordinates

The pipeline flow UI schema doesn't allow negative coordinates for comment objects. This should be allowed.

Fix app_data attribute types in pipeline-flow-ui-schema.json

Following the changes for issue #33 the pipeline-flow-ui-schema.json was found to have some incorrect data types for information contained in the app_data section of pipelineFlow documents. These data types should be corrected.

Allow users to specify if a twisty panel is open by default or not

Add SQL in expression editor supported languages

Expression editor supports SQL language for syntax highlight and autocomplete words. Update this list to include SQL.
https://github.com/elyra-ai/pipeline-schemas/blob/main/common-pipeline/operators/uihints-v3-schema.json#L375

Add a new config option "return_type_label" in function_info

Add a new config option return_type_label to the schema as an optional field that is of type resource_def. In the code we can first check for the return_type_label and default back to return_type if its undefined.

This can be used to translate return column in functions table -

Add new property for multiselect dropdown

As part of elyra-ai/canvas#452, we need to add a new property to indicate a multiselect dropdown, for the multiselect control

Extend zoom function to store translation amounts

Schema ID uses api.dataplatform.ibm.com which does not resolve

e.g. http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json

However it does resolve if you use http://api.dataplatform.cloud.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json

pipeline-schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json

Lines 1 to 5 in 6da3bf0

	{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"description": "WDP Pipeline Flow Schema",
	"type": "object",
	"id": "http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json",

The pipeline-flow-ui-v3-schema.json does not even seem to be deployed to http://api.dataplatform.cloud.ibm.com and is unreachable.

pipeline-schemas/common-pipeline/pipeline-flow/pipeline-flow-ui-v3-schema.json

Lines 1 to 5 in 6da3bf0

	{
	"$schema": "http://json-schema.org/draft-04/schema#",
	"description": "WDP Pipeline Flow UI Schema. Defines UI-only constructs for pipeline flow documents.",
	"type": "object",
	"id": "http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-ui-v3-schema.json",

Descriptions in pipeline flow should be deprecated

There are a mixture of locations for description properties for different pipeline flow entities. Some are in the pipeline-flow schema and others in the pipeline-flow-ui schema. The most appropriate place for description properties is in the pipeline-flow-ui schema because that is where other user-defined information such as labels is stored.

Description properties in the pipeline-flow schema should be deprecated and the schema user should be referred to description properties in the pipeline-flow-ui schema instead.

Remove limitations on zoom scale (k) value

Ref: elyra-ai/canvas#1047

The zoom scale limitations of min 0.2 and max 1.8, while needed for usability purposes in the UI are not necessary for the schema since any zoom scale value should be allowed.

This change will help the following situation: after a zoom-to-fit action on a pipeline with a large number of nodes a scale value lower than 0.2 can be calculated. When the pipeline is saved with that small scale value it needs to be able to be read into the UI when schema validation is switched on.

Incorrect reference to app_data_def in connections schema

In the pipeline-connection-v3-schema.json the reference from the data_asset field to app_data is

"$ref": "http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow-v3-schema.json#/definitions/app_data_def"

but should be

"$ref": "http://api.dataplatform.ibm.com/schemas/common-pipeline/pipeline-flow/pipeline-flow-v3-schema.json#/definitions/app_data_def"

Add space_ref to pipeline-connection-v3-schema

Optional "link" property under description in uiHints

Elyra-canvas issue - elyra-ai/canvas#1000

Add a new link object under description in uiHints

Update Form schema to include icon and heading properties

Changes to Form as a result of work on elyra-ai/canvas#419

Add op property to the model node definition

Model nodes need to be able to record the node type of the modeling node that was used to create the model. For consistency with other nodes this should be stored in an op property.

additionalProperties in pipeline-connection-v3-schema blocks connection.properties having key/value where value is primitive type

This part of the connections schema is specifying that the properties object is an object but that, also, all additional properties must be objects:

This means that any regular field provided for properties like this will be rejected by schema validation:

However, if all of the fields inside properties are the selves objects then it will pass the validation:

Everywhere else in the schemas where additionalPropeties is used it is specified as a boolean. It should be changed to a boolean in this case which would allow any fields to be added to the “properties” field.

Fix schema for parameterDefs

While working on the schema validator for common-properties in elyra-ai/canvas#446, I found there are a few areas that needs to be fixed. A couple issues were reported from testing with the samples in the test harness.

Fix readmes for new repo

Some of the URLs and wording in the README.md files in this repo need fixing for the new opensource repo.

Support alternative output from exit binding node

Some applications want to direct output from exit binding nodes, for example when they have error information they need to direct to another node.

Support strings in expression builder min and max range

Elyra-canvas issue - elyra-ai/canvas#1017
Currently min and max values in range only support numbers. Add support for strings.

Add nested_panel parameter to schema

All panels will be left aligned by default. Add a configuration option nested_panel under group definition. This is a boolean parameter.

new readonlyTable control

A readonlyTable control was recently introduced in elyra-ai/canvas#359. We need to add this control in the schemas

Complex type controls should support complex type parameters

elyra-ai/canvas#255 added support for nested complex structures. We need to update the schemas to reflect these changes.

Update values array items type in datarecord metadata

Items in values array can have primitive data type -

"type": "Can be a primitive type (string, integer, double, date, time, timestamp, binary, boolean)"

Support resize variables for 'resize nodes' feature

Document app_data.ui_data.sub_pipelines field

V3 Parameter set and parameter schema change and v3 pipeline schema change to reference the new parameter set schema

Propose v3 schema change for Parameter set and parameter.
Also propose v3 pipeline schema change to reference new parameter set schema

Here is the corresponding pull request #63

FYI @matthoward366

Move builds to github Actions