bungeemonkee/transformerizer

A parallel transformation library with a producer-consumer design

Transformerizer

A parallel transformation library with a producer-consumer design

Description

Transformerizer is a simple transformation library that allows you to run multi-step trasnformations in parallel. Each step can run in parallel with each other using a producer/consumer design and with itself by setting a differnet number of threads to process each step.

Installation

The easiest way to install this is using nuget: https://www.nuget.org/packages/Transformerizer

Usage

Usage is simple: any enumerable can begin a transformation by calling the BeginTransform(..) extension method. Then call ThenTransform(...) as necessary followed by EndTransform(...).

var enumerable = new int[] { 1, 2, 3, 4, 5, };

var result = enumerable
    .BeginTransform(x => x * 5, 1)
    .ThenTransform(x => x / 3, 2)
	.EndTransform();

Bugs And Feature Requests

Any TODO items, feature requests, bugs, etc. will be tracked as GitHub isues here: https://github.com/bungeemonkee/Transformerizer/issues

transformerizer's Issues

Support ignoring null results from transformations

Sometimes processing an input results in a null. There should be an option to ignore these results (instead of passing them along to the next buffer).

System.DivideByZeroException in TransformerStatistics constructor

Not sure where it is exactly. Probably if there were no results at all from the transform chain.

Add BeginTransformMany

TransformMany(...) is very useful for transformations which can return zero or more results from a single input but there is no analog to being a transformation chain with. Create an analogous BeginTransformMany(...) function.

Support multiple transformations from a single input

Sometimes multiple actions must be taken on a single input (sync a single change to several places, etc.). It would be great if we could support this in a fluid way. Some syntax like this maybe:

var threads1 = 5;
var threads2 = 10;
var items = new List<int>();
var results = items.BeginTransform(threads1) // This would just begin the transformation, essentially creating the transformer
    .Transform(SomeTranformFunction) // Transform the int with one method
    .Transform(AnotherTransformFunction) // Transform the int with another method, this would have to return the same type as the method above
    .ThenTransform(threads2) // This would aggregate the results from the two methods above and create the next transformer
    .Transform(SecondTransformerMethod)
    .EndTransform();

Support type discriminators

Add some syntax to enable a single transformer/threadpool to handle multiple types each with a different transformation function.

Support ending transformations with methods that return void

TransformerExtensions.EndTransformAsync() May not dispose of transformer in the case of an exception

The call to transformer.Dispose() in TransformerExtensions.EndTransformAsync() may not be called in the case of an exception because it is not in a finally block.

Add a statistics framework

Some way to collect statistics without major performance impact could be very useful for debugging. Possible statistics:

• Min/max/average buffer size
• Estimated thread idle time (as percent of total or raw time)
• Total transformation time
• Time when each transformation step completed (as percent of total or raw time)