hardikmaheshwari/legend-engine

Legend Engine module

legend-engine's Issues

RFC: Integrate binding with platform

Overview

A Binding is a concept used to define support in the platform for data in some format. Data is either described using a formal schema or understood through conventions.

Internal to the platform we have the data model to describe the data structurally and instances of it to represent data values. Their external equivalents are the schema (for example a JSON Schema) and the serialized form of the data (for example a JSON document). Models can be generated from schemas and vice versa.

As a part of the development for this RFC, we plan to integrate / refactor bindings into the legend platform so that users can integrate use-cases depending on schemas / data defined in some format

Implementation plan

We plan to tackle the development of binding integration in the following phases.
The ideology behind the implementation plan is to take a depth approach and get simple use-cases working first and then add features to bring feature parity compared to existing support for Flatdata/ Json/ Xsd.

These tasks would be expanded into individual issues containing concrete details.

Phase I: Simple use-cases supported end-to-end with binding

• Finalize & document query semantics and the role of bindings within it
• Fix Binding metamodel and detach it from store hierarchy
• Remove External Format Connection
• Uplift Routing phase to work with both binding & store
• Add support for ->internalize function to work with static string, URL and byte-stream
• Add support for ->externalize function to work with simple use-cases (checked queries are out of scope for this phase)
• Uplift services to work with the binding flow (in absence of mapping/runtime & with byte-stream as a query parameter)
• Consolidate execution plan / nodes for binding flow (includes serialization / deserialization)
• Define testing strategy for bindings
• Studio supports binding queries seamlessly (Support should be at par with queries backed by stores)
• Services with binding queries can be executed via jars.

Phase II: Feature parity compared to existing support for FlatData / Json / Xsd

• Support checked result with externalize
• Uplift binding / externalize function to account for custom configurations

Phase III: Delete code for legacy FlatData / Json / Xsd

• Deprecate legacy code & migrate existing use cases to strategic flow
• Safely delete legacy code

Additional context

1. Binding: https://github.com/finos/legend-engine/blob/master/docs/bindings/bindings.md
2. Query Semantics: https://github.com/finos/legend-engine/blob/master/docs/queries/queries.md