jsam/miniflow

miniflow

Workflow execution based on YAML definitions and NATS streaming.

Disclaimer

This is a work in progress and NOT meant for any production use!

Features

• Simple definition of workflows using YAML
• Easy to setup and execute a workflow
• Designed with ETL in mind
• Supported distributed processing of workloads with NATS streaming messaging.

miniflow's Issues

Question: Synchronization and Fault-tolerance

first of all, I think this project is super awesome. I like that everything is basically in one file and the golang syntax slowly grows on me :)

My first question in about fork/join (or fan-out, fan-in as named in the code, if I understand correctly):
I am looking at at the fanIn function: if I understand correctly, this is invoked whenever a token is received through go channel which is, I soppose, connected through nats? My next assumption is that the "WaitGroup" allows us to synchronize tokens, correct? How do we need to think about this in a miniflow cluster? Will the nats kind of "shard" the tokens belonging to the same workflow intance that they go to the same miniflow instance? Or are you thinking about a distributed locking service? - I am just interested, this is the kind of stuff I like to think about :)

Second Question: assume that te processing of a token fails (program is killed, whatever). Will the nats then redeliver the message and by this the system will "re-try"?

Current Status

Hey Sam, really interesting project, just wondering what the current status of this is? did you move to something else? interested to know.