absaoss / enceladus Goto Github PK
View Code? Open in Web Editor NEWDynamic Conformance Engine
License: Apache License 2.0
Dynamic Conformance Engine
License: Apache License 2.0
Source may contain Timestamp data as milliseconds since the Unix Epoch:
1528797510650 = 2018/09/12 09:58:30.650
As this is not a standard SimpleDateFormat there would need to be a custom Stadardisation specific string, e.g.: UNIXEPOCH
Provide a mechanism to convert native copy book to Spark Struct so this can be uploaded and managed directly in Menas.
Copybook contains too much information to easily store in Spark Struct schema, maybe we should rather store it in Mongo as an uploaded attachment.
With the new UI, all changes even micro changes are incurring multiple versions, this is good and bad as we have a full roll back on any change to a very granular point.
The proposed idea, is to have a separate version system for tracking changes in Menas (when creating and updating definition) compared to version of the dataset we run when doing spark-submit, we will have too many versions due to micro-changes that Menas is going to introduce
If a user clicks edit over Basic Info of any of the models (Schemas, Datasets, Mapping tables), changes description, and then clicks cancel, the change still gets shown in the current window. It almost seems as the update went through.
This goes away after window refresh and does not send any call to database or such.
We need to prioritise stability for version 1.0.0.
To do this, we need to make up for missing unit, integration and system tests.
Have a view of each entity so that we can see historical versions (ie those that have been "deleted")
Users should be able to search filter and sort their schemas/mapping tables/datasets in the Menas UI for easier navigation.
We need to add integration tests for the Menas Rest API.
Handle string based formatting that source may output e.g.:
"1,000.00" = 1000.00
We need to improve coverage of our unit tests in the conformance module.
Array transformations are expensive because we explode and collapse arrays for each conformance rule operating on array columns. We could improve performance by merging rules that operate on the same array columns and decreasing to a single explosion-collapse.
The UI should validate that whitespaces are not present in HDFS paths, that those paths exists, etc.
Dataset/Mapping table/Schema names should not be allowed to have whitespace as that can cause issues
Currently, entity definitions in Menas cannot be exported. The only thing you can export is a file uploaded for a Schema entity.
It would be useful if users could export and import any entity definition as a json.
An open question is how to handle conflicts (cases where a dataset definition with the same name and version already exists). This conflict resolution should be drawn up in a shared doc and once resolved posted here.
This task is similar in nature to #594, but meant for manual single item import/export, while #594 is meant for direct promotion via HTTP and in bulk.
Menas should be able to validate that the specified default schema values are valid fo the associated data type.
Post Conformance result schema should be published back into Menas so we have inbound and outbound data schema
Maybe turning Menas into a central schema store and allow harvesting from other tools
The unit tests should look like this:
Conformance rules should have a more elaborate description.
We need to improve coverage of our unit tests in the utils module.
We lack scaladocs on most APIs, would really help to improve documentation.
This involves the reduction of parameters submitted to the spark for both Standardization and Conformance. Instead these will be provided via Menas.
This should greatly improve the platform's integration capabilities, e.g. with scheduling tools like Oozie.
We need database integration tests to check all database interactions Menas can perform.
This involves for each test case:
The dataset should have a runs page showing run metadata, as per old Menas.
Reminder: add thousands separator.
This involves Cobrix integration in Standardiation.
We can implement a set of routines that provide schema difference give 2 schemes to track schema changes between versions. That may include:
But this will require some UI work as well to visualize the difference. Probably the easiest way is to display the above lists as text boxes.
We could put a warning on views that have entity references pointing entities with old versions.
Example:
We need a view in Menas to add/update/delete/reorder Conformance Rules.
Currently we have no way of ensuring that the models used in Enceladus (std and conformance) are compatible versions with the ones stored in Menas. We can add a header to the HTTP requests that says what model version Enceladus is using, then Menas can check compatibility and respond accordingly.
If the header is not specified Menas should ignore the compatibility check to avoid breaking compatibility for people using Encealdus as a library.
We need to improve coverage of our unit tests in the standardization module.
Ensure that if the input to standardization contains the pre-defined error column name, the schema of the field matches our predefined schema, otherwise fail.
User and admin roles need to be differentiated. User Read/Write permissions need to be restricted to either their own entities or there could be some sort of group restrictions.
We need to set up real e2e system tests which include everything
Do not write parquet file with attributes not registered in the schema, this is usually a symptom where source has attributes, but the schema in the schema repository has not been updated.
Want to use this to ensure owners keep up with schema changes and give control of data distribution for attributes to data owners
Provide 1 JAR artifact for both Conformance and Standardisation.
This will simplify configuration files and also reduce risk of running mismatched STD and CONF versions by users.
Enable an efficient and clean way to implement rerun and also expected override capability for data loads (ie End of Month, End of Day and etc)
Currently conformance allows for 0 field schemas to be inserted and used, this is only picked up when running Enceladus. This is usually an error with schema definition generation or manual manipulation
Currently disable calls calls to the new rest API use HTTP GET requests, but GET should be idempotent. Instead we should use HTTP DELETE.
POST to create entity.
PUT to update entity.
I would like to suggest that there be a different folder structure or way of searching on Menas.
I think it would be good to have a folder per source system and then the tables that belong to that source will be in there instead of there just being a list of tables with no indication of which source system they are from.
Consider performance optimisation with respect to wide data (tables with thousands of columns). The options are discussed at Spark DEV Mailing Lists http://apache-spark-developers-list.1001551.n3.nabble.com/Performance-Spark-DataFrame-is-slow-with-wide-data-Polynomial-complexity-on-the-number-of-columns-is-tt24635.html
Allow users to capture the following information for dataset level definition:
Frequency definitely could feed into our discussion for integration with Oozie and schedulers
Currently we recommend user populate as additionalInfo metadata in INFO file, which should live in Dataset definition rather than "run" metadata (INFO file)
Users should be able to construct and validate their Schemas through the Menas UI.
We have two separate ValidationExceptions:
Good design dictates a clear separation between the two or a unification.
There is also some code duplication in conformance rules, specifically aimed at validation, that would best be extracted to a common location.
It would be helpful to add some performance metrics to Run screen, such as:
An input data size and an output data size should be provided by backend. All other information can be derived from control checkpoints i guess.
There is a generic key/map pair in _INFO file. I guess we can use that to provide information about size of raw, standardized and published datasets. Also, possibly it should include information about standardization and conformance elapsed time.
The raw folder size, std folder size and conformed folder size can be saved in metadata fields of an _INFO file and Run object during std/dynconf jobs execution.
This involves creating an example module with an implementation of a Spark job using the enceladus-conformance as a library to implement a CustomConformanceRule and its respective interpreter.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.