The efbt from eclipse

update oomph profile

update oomph profile to point to the eclipse/efbt repositiory, and to include Mylin egit integration feature

Add automated tests

Add automated tests, probably using JUnit, we should be able to run these with Jenkins CI.
The initial focus should be on the functions of the classes in the org.eclipse.efbt.model.util package

use the model repository component

Calls to create attribute lineage models, or dat lineage models should register the results with the model registry component

refactor model into high level components

update website at www.eclipse.org/efbt with user focussed overview

update website at www.eclipse.org/efbt with user focussed overview , Wiki will be used for project develolopement focussed information such as design and build and contribution guidelines.

Add a web service for querying the data lineage model

Add a web service for querying the data lineage model.
We may want to consider GraphQL

add wiki for running the code generator

update == function to have name EqualTo

This change is already incorporated but the file getAttributeLineageForJustOneCell.platform_call should have been regenerated and checked in.

Update Wiki with How-to: Visualization of an Attribute Lineage Model instance for just one report cell

Update Wiki with How-to: Visualization of an Attribue Lineage Model instance for just one report cell

add incremental processing with lineage

add incremental processing.
Incrementatal processing is where we have already processed may transactions, and then we add one (or some) more, and redo processing. The goal is to only recalculate what needs to be recalculated after one increment, to enable realtime or near time calculation and efficient what-if processing.
This Issue focusses on the lineage aspect of this, along with how models are affected, and aims to introduce a model of computation for incrementatl processing which should be favorable to fast incremental processing.
A future issue will focus on optimising the performance of incremental processing.

create maven build for EFBT project.

We should create a maven build for the project.
Note that currently org.eclipse.efbt.parser.vtl and org.eclipse.efbt.importer.birdaccessdb both contain links to external jars, and this should be addressed in the maven build.
A maven build will help us intergrate with Continous integration also, and so enable more automated testing.

add ecore genmodel to xsd model to simplify getting started

we will add a .genmodel in the org.eclipse.efbt.model project's xsd_model directory, so that users can get started quickly with limited ecore knowledge by generating all java classes from this file after installing the dev environment and git project using oomph.

Update Wiki with example of initial usage

Add to Wiki an example of running the code generator for an example, getting a subset of the attribute lineage, supplying some input data, and creating results with data lineage.

Add a WebService API

Eclipse Free BIRD Tools follows a Model View Controller architecture.
Currently the View uses the desktop-only components which are Eclipse RCP and Eclipse Sirius.
We wish to make the functionality of Eclipse Free BIRD Tools available via a webservice, so that another view layer (for example a web based application) can access the model in a useful way to provide its own visualization, and to be able to make requests for the model to change.

Update Wiki with How-to: compare Attribute Lineage Models

Data Lineage Model is not always showing the correct Function Spec

Improve Code Generator to Remove unnecessary files

some .about files have the wrong date

add wiki for running importing and viewing the BIRD artefacts from MSAccess

Load input data from an external source

The input data for the Attribute Lineage Model, like a specific 5% loan and a AAA counterparty, is currently used to generate code also, so the code generator will actually create a Scala class with data initialised which represents that input data. This is fine for test data which may include 10 transactions, but does not scale well to millions of positions in a production environment. we wish to change this to get data from an external source, perhaps from a relational database or more common input format.

Add a web service for querying the Attribute lineage model.

Add a web service for querying the Attribute lineage model.
We may want to consider GraphQL

Add Code which parses VTL Transformations according to VTL grammar

Improve visualisation of Lineage Models

put model documentation on the web

make regulatory text multi-line

Add Platform Call to get the Attribute Lineage Model for just one report cell

Add a Platfrom Call to get an Attribute Lineage Model for just one report cell. The Platform call should be provided with a reference to the Attribute Lineage Model for full processing and a reference to the relevent cell that we want to have a subset of the Attribute Lineage Model for.

Improve visualizations

remove reference to external jars in build configuration

Currently there are 3 jars that need to be added manually to the build path, this should be simplified when we move to a maven build by addressing issue #6 .
Currently we link to a specific directory on my c: drive, so this needs removed and the build instructions on the wiki will cover how to add these 3 jars to the build path.

Create Attribute Lineage Diagram from BIRD artefacts

Currently we can create an AttributeLineageModel Instance from model instances (stored as XML files) which represent programs in Traceable Reporting Language (TRL).

TRL uses SMCubes like BIRD does, so many artefacts are the same (e.g. dimensions, domains, variables, cubes). The part of TRL which deals with transformations is extremely similar to VTL, TRL stores its transformations in a node structure very similar to the node structure produced by parsing VTL programs according to the VTL grammar (TRL has the same concepts of keep, rename, group by , calc, etc) .

Currently we use the TRL nodes of a TRL 'program' to generate an instance of an AttributeLineageModel, using the Platform call GetAttributeLineageModel which uses the utility classes in org.eclipse.efbt.model.util (particularly AttributeLineageUtil.java).

We need to change this so that TRL is not used at all and can be deleted, and instead use the description of VTL transformations (along with the descriptions of variables, domins etc) to create the AttributeLineageModel. The first step in doing this is parsing the VTL Transformation into nodes, then using this structured version to create an AttributeLineageModel programmatically, most likely using the visitor pattern,

update wiki with getting started an initial build instructions

update wiki with design overview

update oomph profile to use latest version of Eclipse

add documentation for functional modules

add Tooltip to Columns on Attribute Lineage Model visualization

add Tooltip to Columns on Attribute Lineage Model visualization.
Currently the columns on the Attribute Lineage Model visualization show variable names.
However it is useful to also have acces to the variable ID via the tooltip.
This is because when we visually drilldown into functions or RowApproaches these typically use the varaible id (like ECB2_ENTITY_LEI) which is different from the name (like Entity LEI )

refactoring of class names in Model

To make the lineage models clearer we should rename some classes to make the model clearer.

1.) We should rename DerivedCell as FormulaCell, since it is the same concept as a cell in a spreadsheet which contains a formula.
2.) We should rename RowFunction as RowCreationApproach , since it determines the approach like 1to1, groupby, join etc for creating rows.
3.) We should rename CubeLogic as RowCreationApproachForCube
4.) we should note that Column function is already well named.
5.) We should rename FunctionalRowLogic as CubeTransformationLogic as it contains the RowCreationApproach, and the Columnfunctions

We should consider if we should rename some of the containing EPackages, and what impact this wold have. It would make sense for data lineage package to be renamed cell_transformation_logic, and some of the classes to moved around between row_transformation_logic and cube_transformation_logic

eclipse / efbt Goto Github PK

efbt's People

Contributors

Stargazers

Watchers

Forkers

efbt's Issues

Recommend Projects

Recommend Topics

Recommend Org