v12technology/fluxtion

core fluxtion static event processor generator tool and runtime components

fluxtion's Issues

Sliding window support

add sliding window support to streaming processing

Compiler code tidy

remove generic event and signal they are unused
Typos
Encapsulate all fields in SEP Config
Remove declarative config - just add factories
Remove MODEL_EXTENSION from velocity template context

upgrade velocity to velocity-engine-core latest

upgrade velocity to velocity-engine-core latest

velocity was last released in 2010, transitive dependencies have vulnerabilities

Re-entrant event support

Ensure the graph can handle re-emtrant events by queueing and then dispatching

NodeNaming and FilterNaming to be service loader based

NodeNaming and FilterNaming provide ways to customise the source code generation, currently just one class can be registered. requirements:

• Multiple instances registered
• ServiceLoader mechanism used to extend the registered classes
• Add priority for a strategy so users can alter the resolution order
• Keep progressing through registrations until no more are left then use platform default strategy

Stream like graph building

Provide a fluent coding style for building processing graphs. Should resemble Java streams api as much as possible to reduce learning curve for developers

Multiple guards for OnEvent(dirty=true) callaback not working

Test to be added

When annotation a callback method for onevent notification we can specify to invert the boolean guard conditions. This works for scalar fields by adding a ! to the guard, but for multiple parents this is not working.

The generated sep should have each guard prefixed with ! for OnEvent(dirty=true)

Add support for zero validation rules in CSV parser

The com.fluxtion.ext.futext.builder.csv.RulesEvaluatorBuilder builds rule processing logic for a monitored node, three types are supported:

• Normal node
• WrapperNode
• RowProcessor

The case for zero rules need to be handled correctly for all three. Currently only BuilderRowProcessor nested class supports the zero rule case.

CSV documentation

The CSV module needs clear documentation with examples. This will help adoption of the library by new users.

GroupBy support in stream

Build groupBy support into stream api for:
normal aggregte
sliding window aggregate
tumbling window aggregate

CSV multi-dispatch

Use a single file to drive multiple parsers.

The input file schema:
col_0 = simple name of class
col_n = data fields for the parser type

The chars upto the first delimiter will be swallowed by the dispatcher and then pushed to the matched marshaller. The marshaller instance will register with the multiplexing dispatcher with the marshall class type:
dispatcher.addMarshaller(EventHandler marshaller, Class marshallClass)

A sink can be added as the target of the any incoming message
dispatcher.addSink(EventHandler sink)

CSV lookup values

provide a method for deriving field values in a target bean from a lookup value.

Potentially add validation post the lookup, the validator could infer when to be applied based on the context.

Possibly combine with #10 to ignore the derived value

lifecycle support re-entrancy

Thread safety of the maven plugin

Building this project via mvn clean install -T 1C throws this warning:

[WARNING] *****************************************************************
[WARNING] * Your build is requesting parallel execution, but project      *
[WARNING] * contains the following plugin(s) that have goals not marked   *
[WARNING] * as @threadSafe to support parallel building.                  *
[WARNING] * While this /may/ work fine, please look for plugin updates    *
[WARNING] * and/or request plugins be made thread-safe.                   *
[WARNING] * If reporting an issue, report it against the plugin in        *
[WARNING] * question, not against maven-core                              *
[WARNING] *****************************************************************
[WARNING] The following plugins are not marked @threadSafe in fluxtion :: examples :: docs.core:
[WARNING] com.fluxtion:fluxtion-maven-plugin:1.7.25-SNAPSHOT
[WARNING] Enable debug to see more precisely which goals are not marked @threadSafe.
[WARNING] *****************************************************************

If the plugin is indeed thread-safe, can you mark it as such please ?

CSV add empty value

Specify a value to use if the field in the CSV file is empty. This should be a CharSequence and will be set before any converter or validation rules are executed.

GroupBy sliding window support

Sliding windows for groupBy constructs

Move extension and integrations projects from Fluxtion to incubator

Move these projects from fluxtion to incubator:
extensions - streaming
extensions - text
integrations

Changes file

Is it a good idea to maintain a changes file? If so can we automate it with the issues/project board provided by Github

How to attach issues fixed to release notes

Want to automatically generate release note that gathers together all issues on github that have been closed in a release. Please help

Performance improvement for interpreted event processor

Issue #88 introduced support for interpreted event processing. Currently this is implemented with reflection based dispatch, which is less than optimal.

Goals:

• Improve performance with code optimisations
• Introduce a JMH test harness to measure before and after
• Record results for an article
• ZeroGc
• Response time as close as possible to static compiled event processor

Pluggable log implementation for MsgBuilder

Enable different log implementations to be injected in the logging framework in ext::declarative.

In LogBuilder an AsciiLogger is added by default. Allow different implementations to be substituted at runtime.

Inprocess compiling of SEP

The Fluxtion event stream compiler is designed to be used out of process. This can be inconvenient as it requires several pom execution definitions. Users want to get started as quickly as possible, introduce a programmatic interface that gets developers up and running more quickly.

Add to the generator project, create an InProcessCompiler utility, things to consider:

• generate only
• instantiate a handler
• Potential class loading ./ state problems
• configure namespaces and directories

Ensure that all building logic is kept in the builder/generator class space so that a generated SEP can be used only with the api's on the classpath.

Better landing page

The landing page/readme is poor at explaining what FLuxtion does and the benefits it offers. Start a discussion to get ideas about what are the best themes to publicise and the how to structure the landing page

Validation logging

Is your feature request related to a problem? Please describe.
Provide a means of trapping and recording all validation errors. Fluxtion injects Validation Loggers into validators and RowProcessors. Gather all the logs together for displaying to user to fix errors

Describe the solution you'd like
Row/field level logging
Cause of failure
Configurable output location/format

Additional context
possibly inject collated Log into commodity logger(slf4j etc.) for rendering/formatting etc

CSV ignore transient bean properties

At present all bean properties are mapped to columns in auto bean mapping. If any columns are missing an error is logged and thrown.

Derived values will not be supplied in the source data but are required in the final bean, the marshaller should ignore missing derived fields from the source.

Proposal
Fields marked with transient keyword will be ignored by the marshaller missing columns from source data will be acceptable.

Refactor package names to reflect module structure

A refactoring of modules has left the packages out of sync with the updated structure. Update packages to match

CSV - shema based generation

Is your feature request related to a problem? Please describe.
Allow non programmers to specify a csv marshalling process in an idl. Enabling wider adoption of Fluxtion processing.

Describe the solution you'd like
user provides a definition similar to: csv lint or json table schema
Fluxtion processes CSV description and generates two outputs:

• A self contained executable jar reads CSV, validates and potentially dump to DB
• A fluxtion event processor component that can be used in a larger ETL process

Features to include:

• required/non required
• type
• converters
• validation methods
• lookups
• derived calculations
• max/min col width
• default values
• patterns for regex match and date formatting
• all standard CSV processing parameters: lione endings, separator, quoting etc.

The schema will generate a bean with annotations and a set of processing instructions to build the Fluxtion SEP description

Describe alternatives you've considered
There are no other zero cost abstraction parsers I can find, but another nice parser univocity

Additional context

Correct groupId to deploy to maven central

The group ID for the maven poms was changed to:
<groupId>com.fluxtion.**core**</groupId>

Maven central only allows fluxtion to deploy with:
<groupId>com.fluxtion</groupId>

update all poms and create new release

Remove integration tests in generator project

Integration tests were originally used within generator project as inline compiling was not supported by Fluxtion. Many tests have been migrated to use com.fluxtion.generator.util.MultipleSepTargetInProcessTest.

MultipleSepTargetInProcessTest has the following advantages:

• Supports in-process compiling no need for integration tests
• Parameterized to test both compiled and interpreted versions produced by compiler

Optimise imports on generated files

Optimise the imports of generated files,. Currently the generated output Java files have unused imports. Find a solution that can:

• analyse the source file after generation
• remove any used imports
• rewrite the source file with redundant imports removed
• no incomplete file writes allowed, must be transactional

Compiler does not work with Java 12

The fluxtion compiler does not work with Java 12. Fluxtion uses the compiler from chronicle which uses deprecated api's in Java 11, that are removed in Java 12.

To repeat run Fluxtion build in Java 12.

Need a solution with custom class loader and source to Java compiler. Ideally should work as part of the Java Module System.

Support primitives in event streams

Zerogc charsequence to primitive

integrate one of the zerogc libraries into the CSV marshaller that converts charsequence to primitive:

• jnumbers
• plexus utils

Annotation based factory registration

develop a methodology of registering factory xml files for processing by Fluxtion compiler using annotations similar to the @SepBuilder annotation in release 1.6.3

Move count function to api

the count function resides in builder, move impl to api, implement static creation function in builder

Tests fail on JDK >=9

Describe the bug
The tests fails on OpenJDK 9 and higher

To Reproduce
see https://travis-ci.com/Allsimon/fluxtion/builds/142587406

Add Java 17 compatbility

the openhft runtime compiler does not support Java 17 by default. requires --add-opens directives at runtime. Replace with a simpler in memory compiler as th caching of classes is not required atm

Remove builder module

Remove the builder module and move classes to api and generator modules.
rename modules:
api -> runtime
generator -> compiler

Yaml event processor builder

use a yaml description to define and build an event processor

CSV multiple definitions feed into single dispatcher

Extending multiple marshallers and connect multiple marshallers to a single DispatchingCsvMarshaller. One definition will create a single dispatcher handling a CSV data source containing heterogeneous events. The output event can be pushed to a registered EventHandler that acts as a sink for all events.

Annotation processor generation

Add annotation processor functionality to the compiler project to generate and build an event processor at build time

Add Key type to GroupBy map

GroupBy gives access to a Map in the generated event processor. The map key has no generic type, and is defined as:

Map<?, Wrapper>

update to:

Map<K, Wrapper>

Contributors file

Create a contributors file:
We want people to join in, so need to make this clear and simple entry point.

Need to have a copyright assignment process that works for contributors.

CSV Multiple marshaller definitions in a single file

Describe multiple CSV marshaller definitions in a single file would be a time saver for developers. Presently requires multiple pom execution definitions, this will dissuade developers from adopting the framework.

leverage in process compiling and add multiple configs in a single pass.

Documentation for streaming extension

There is no documentation for streaming extension.

Streaming offers significant functionality similar to java streams, but works for reactive processing graphs. It is a very nice feature that new users will probably want to use before anything else. The documentation should be easy to use and understand.

The landing page should include a streaming example to get people up and running.

Documentation is in markdown format hosted by gitbooks current doc

Method reference for test

Update class based test descriptions in TestBuilder to be specified with method reference. This has been completed for instance tests, need to update for static test methods.

for example change the signature:
public static <T extends Test, S extends NumericValue> TestBuilder<T, S> buildTest(Class testClass, S n)
to
public static <T extends Test, S extends NumericValue> TestBuilder<T, S> buildTest(Class testClass, Function<T, S> S n)

the first example looks for a single abstract method

Stream like wrapper

many functions in Fluxtion return a wrapper, add stream like functions to the wrapper to operate on the underlying instance, candidate methods:

• filter
• map
• function (transform)

Interpreted dispatcher

Currently only a code generated dispatcher is supported by Fluxtion. As the size of graph increases the compiler cannot effectively optimise methods that are above a certain size, causing a performance degradation.

Provide an option that stores the dispatch tree as an in memory data structure, such as an array, Each element in the structure encapsulates the invocation to a user class, and meta information about dependents.

A further advantage this may allow dynamic graphs to be more easily accommodated when implemented.

Performance notes:
Investigate using LambdaMetadataFactory to generate efficient invocations
A bitset is a good structure for storing clean/dirty notifications of nodes. If parent node changes it can set the dirty bit for child nodes

Sliding window and groupby combined

Sipporrt sliding windows for a groupBy construct

All generated artifacts to have license header

All generated artifacts should have the license header present. For example some of the generated classes from ext.declarative.builder do not add the license header.