rec-framework/rec-core

Data pipelining service

Rec

Rec is a simple framework to check and transform data record files(like CSV). It provides several DSL to describe the relationships and transformation rules to check the data correctness and integrity.

Rec v2 is currently under design, and you can see website for details.

rec-core's Issues

MalformedInputException: Input length = 1

When access an ascii file.

Migrate unit test to different modules and replacing with Java

Currently following test need to be migrated:

• rec-accessor and record wrapper
• rec-sepval parser
• all rec-core test need to be migrated to Java

Fluquent API using operator overloading?

http://2ality.com/2011/12/fake-operator-overloading.html

Replace hand-crafted logger with SLF4J

Currently rec-core using self-created logger util, and we need replace it with SLF4J which might be better integrated with other logger facility.

JSON.stringify for POJO

First line output as "." in version 1.3

RT.

Initializer bug on Windows

[EPIC]Rec#reactive

Sub issues or PRs:

• #44 require('rx') issue.

Reduce dependency by using Java instead of kotlin

RT.

Performance enhancement by add indecies for records

Performance issue found when using JS scripting tools:

• RecSet#contains takes several seconds to check A contains B, when A has about 200,000 records.

Proposed using indices

Travis CI cannot run checkstyle with Java 8 syntax by default

As showed in #39 .
Raised this issue to fix it.

Replace Kotlin script engine with JavaScript

Currently using kotlin has a lot of weird behavior, try switch JS instead.

As this is built on top of Java 8, an option is nashorn.

but not sure nashorn support for modern ES standard, still will using Rhino.

[EPIC]Restartability

• Fail fast
• then restart.

Subissues and PR:

#45

FlatFileTarget IndexOutOfBoundException

code:

const cnt = counter(function() true);
const file = flat('timestamps.csv');

csv('combined.csv', "...")
   .tee(cnt)
   .to(file);

file

blabal, ... ,,,

Caused by: java.lang.IndexOutOfBoundsException: Index: 13, Size: 13
        at java.util.ArrayList.rangeCheck(Unknown Source)
        at java.util.ArrayList.get(Unknown Source)
        at net.kimleo.rec.sepval.SepValEntry.get(SepValEntry.java:19)
        at net.kimleo.rec.sepval.SepValEntry.get(SepValEntry.java:7)
        at net.kimleo.rec.v2.accessor.RecordWrapper.getByIndex(RecordWrapper.java:41)
        at net.kimleo.rec.v2.accessor.RecordWrapper.get(RecordWrapper.java:34)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.collect(Unknown Source)
        at net.kimleo.rec.v2.model.impl.FlatFileTarget.put(FlatFileTarget.java:30)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
        at java.util.Iterator.forEachRemaining(Unknown Source)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Unknown Source)
        at java.util.stream.AbstractPipeline.copyInto(Unknown Source)
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source)
        at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
        at java.util.stream.ReferencePipeline.forEach(Unknown Source)
        at net.kimleo.rec.v2.model.impl.FlatFileTarget.putAll(FlatFileTarget.java:35)
        at net.kimleo.rec.v2.model.Source.to(Source.java:23)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:126)
        ... 15 more

Rec v2 API Specification & Guide

Discovered by #10 as we changed the API and architecture

Performance issue: Reduce memory cost for records

• For release 0.1.0, it will cost more than 1 GB heap size when only 100,000 records.

need to be optimized.

Rec cannot use RxJS because cannot initialize global context

Rec get following error while loading and running rx.js:

=> Rec v2
Exception in thread "main" org.mozilla.javascript.JavaScriptException: Error: RxJS could not find any global context (window, self, global) (rx#34)
        at org.mozilla.javascript.gen.rx_3._c_anonymous_2(rx:34)
        at org.mozilla.javascript.gen.rx_3.call(rx)
        at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:32)
        at org.mozilla.javascript.gen.rx_3._c_anonymous_1(rx:20)
        at org.mozilla.javascript.gen.rx_3.call(rx)
        at org.mozilla.javascript.optimizer.OptRuntime.call2(OptRuntime.java:42)
        at org.mozilla.javascript.gen.rx_3._c_script_0(rx:20)
        at org.mozilla.javascript.gen.rx_3.call(rx)
        at org.mozilla.javascript.gen.rx_3.exec(rx)
        at org.mozilla.javascript.commonjs.module.Require.executeModuleScript(Require.java:342)
        at org.mozilla.javascript.commonjs.module.Require.getExportedModuleInterface(Require.java:290)
        at org.mozilla.javascript.commonjs.module.Require.call(Require.java:219)
        at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:63)
        at org.mozilla.javascript.gen.abc_js_2._c_script_0(abc.js:1)
        at org.mozilla.javascript.gen.abc_js_2.call(abc.js)
        at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:393)
        at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3282)
        at org.mozilla.javascript.gen.abc_js_2.call(abc.js)
        at org.mozilla.javascript.gen.abc_js_2.exec(abc.js)
        at org.mozilla.javascript.commonjs.module.Require.executeModuleScript(Require.java:342)
        at org.mozilla.javascript.commonjs.module.Require.getExportedModuleInterface(Require.java:290)
        at org.mozilla.javascript.commonjs.module.Require.requireMain(Require.java:137)
        at net.kimleo.rec.v2.scripting.Scripting.runReader(Scripting.java:34)
        at net.kimleo.rec.v2.scripting.Scripting.runfile(Scripting.java:22)
        at net.kimleo.rec.App.execute(App.java:66)
        at net.kimleo.rec.App.main(App.java:28)

Discovered when spike #34

Design Consideration: Splunk Search Processing Language

see: https://www.splunk.com/content/dam/splunk2/pdfs/solution-guides/splunk-quick-reference-guide.pdf

[EPIC] Rec-cloud requested features

Features:

• CSV loader accepts Reader/InputStream instead of File.
• Scripting.runScript accepts Reader/InputStream instead of file name.
• ExecutionContext / 'rec' module should only shared by every scripting instance instead of static/global.
• Add Resource interface which provides Source directly.

Enhancement: using Record#toString to get the original record.

This is useful when trying to keep original record without specify all the field names

Google DSPL support

https://developers.google.com/public-data/

Add loader and exporter for DSPL format.

Loader issue on Windows

Loader cannot get data file using rec file pattern

Add encoding convertor?

Issue found in #35 , when specified CSV file in utf-8 mode, it will cause potential encoding problem.

Suggest add a conversion method to convert file from one encoding to another.

Add specification document for DSLs: accessor, query selector, etc.