danielgerlag/workflow-core

Lightweight workflow engine for .NET Standard

Workflow Core

Workflow Core is a light weight embeddable workflow engine targeting .NET Standard. Think: long running processes with multiple tasks that need to track state. It supports pluggable persistence and concurrency providers to allow for multi-node clusters.

Announcements

New related project: Conductor

Conductor is a stand-alone workflow server as opposed to a library that uses Workflow Core internally. It exposes an API that allows you to store workflow definitions, track running workflows, manage events and define custom steps and scripts for usage in your workflows.

https://github.com/danielgerlag/conductor

Documentation

See Tutorial here.

Fluent API

Define your workflows with the fluent API.

public class MyWorkflow : IWorkflow
{
    public void Build(IWorkflowBuilder<MyData> builder)
    {    
        builder
           .StartWith<Task1>()
           .Then<Task2>()
           .Then<Task3>();
    }
}

JSON / YAML Workflow Definitions

Define your workflows in JSON or YAML, need to install WorkFlowCore.DSL

{
  "Id": "HelloWorld",
  "Version": 1,
  "Steps": [
    {
      "Id": "Hello",
      "StepType": "MyApp.HelloWorld, MyApp",
      "NextStepId": "Bye"
    },        
    {
      "Id": "Bye",
      "StepType": "MyApp.GoodbyeWorld, MyApp"
    }
  ]
}

Id: HelloWorld
Version: 1
Steps:
- Id: Hello
  StepType: MyApp.HelloWorld, MyApp
  NextStepId: Bye
- Id: Bye
  StepType: MyApp.GoodbyeWorld, MyApp

Sample use cases

• New user workflow

public class MyData
{
	public string Email { get; set; }
	public string Password { get; set; }
	public string UserId { get; set; }
}

public class MyWorkflow : IWorkflow
{
    public void Build(IWorkflowBuilder<MyData> builder)
    {    
        builder
            .StartWith<CreateUser>()
                .Input(step => step.Email, data => data.Email)
                .Input(step => step.Password, data => data.Password)
                .Output(data => data.UserId, step => step.UserId)
           .Then<SendConfirmationEmail>()
               .WaitFor("confirmation", data => data.UserId)
           .Then<UpdateUser>()
               .Input(step => step.UserId, data => data.UserId);
    }
}

• Saga Transactions

public class MyWorkflow : IWorkflow
{
    public void Build(IWorkflowBuilder<MyData> builder)
    {    
        builder
            .StartWith<CreateCustomer>()
            .Then<PushToSalesforce>()
                .OnError(WorkflowErrorHandling.Retry, TimeSpan.FromMinutes(10))
            .Then<PushToERP>()
                .OnError(WorkflowErrorHandling.Retry, TimeSpan.FromMinutes(10));
    }
}

builder
    .StartWith<LogStart>()
    .Saga(saga => saga
        .StartWith<Task1>()
            .CompensateWith<UndoTask1>()
        .Then<Task2>()
            .CompensateWith<UndoTask2>()
        .Then<Task3>()
            .CompensateWith<UndoTask3>()
    )
    .OnError(Models.WorkflowErrorHandling.Retry, TimeSpan.FromMinutes(10))
    .Then<LogEnd>();

Persistence

Since workflows are typically long running processes, they will need to be persisted to storage between steps. There are several persistence providers available as separate Nuget packages.

• MemoryPersistenceProvider (Default provider, for demo and testing purposes)
• MongoDB
• Cosmos DB
• Amazon DynamoDB
• SQL Server
• PostgreSQL
• Sqlite
• MySQL
• Redis

Search

A search index provider can be plugged in to Workflow Core, enabling you to index your workflows and search against the data and state of them. These are also available as separate Nuget packages.

• Elasticsearch

Extensions

• User (human) workflows

Samples

• Hello World

• Multiple outcomes

• Passing Data

• Parallel ForEach

• Sync ForEach

• While Loop

• If Statement

• Events

• Activity Workers

• Parallel Tasks

• Saga Transactions (with compensation)

• Scheduled Background Tasks

• Recurring Background Tasks

• Dependency Injection

• Deferred execution & re-entrant steps

• Looping

• Exposing a REST API

• Human(User) Workflow

• Testing

Contributors

• Daniel Gerlag - Initial work
• Jackie Ja
• Aaron Scribner
• Roberto Paterlini

Related Projects

• Conductor (Stand-alone workflow server built on Workflow Core)

Ports

• JWorkflow (Java)
• workflow-es (Node.js)
• liteflow (Python)

License

This project is licensed under the MIT License - see the LICENSE.md file for details

workflow-core's Issues

Web front-end to admin/monitor workflows

• Search & query status of workflows
• Suspend & resume
• Terminate

Exception Body in ExecutionError

Hi Daniel,

first of all thanks for that great peace of work. Really really helpful!

I just wanted to ask if it would be possible to extend the ExecutionError model with one column containing the complete exception body and not only the ex.Message.

Despite that, i asked myself, if it would make sense to inject some kind of action processor / provider. That way the tasks could be submitted to one execution thread (one thread that all my other repository actions use) in a controlled manner.

Otherwise, in my understating, the tasks get submitted to the System Thread Pool and needs safety handling.

Does that make sense?

Regards

about the sqlserver

  services.AddWorkflow(x => x.UseSqlServer(@"Server = 117.169.87.18;Database = workflow;User ID = sa;Password = jlkj207.;Trusted_Connection = False;", true, true));

can you show me , where the sql database structure file. how can it work ? and the data will record the every step ? maybe you can give a full example

Mluti-events for same event name/key

I put an event in a loop and it will keep going before it meets a condition.
like this:

builder
    .StartWith<SayHello>()
    .While(data => data.flag == true)
        .Do(x => x
            .StartWith<DoSomething>()
            .WaitFor("MyEvent", data => "0")
                .Output(data => data.Events, step => step.EventData)
            .Then<DoSomethingElse>())
    .Then<SayGoodbye>();

And event may published from multi-source.
It means it may come in very close time.
It tried and it will miss event if it's close enough.
The reason is the 2nd event come before loop go back to WaitFor step.

Is there any idea for this case?

--
Thanks for these.
It's really cool.

Use object serialization for workflow data in MongoDB provider

Currently, it is serialized as a string which makes it difficult to run queries against the persistence store that can evaluate the contents of the custom workflow data

Add Step "Name" to Persistence

How difficult would it be to add the Step Name to the persistence model? We have 16 steps, and are showing them in a dashboard, but "Step 12" doesn't mean anything to the viewer.

It looks like a property needs added to the ExecutionPointer model based on a quick glance?

Back step in Workflow

Hi Daniel
Is it possible to step back in a workflow (like "return ExecutionResult.Back()"; ?)

Best regards
Sigi

How to get next steps in workflow

Hi, i'm using workflow-core as my workflow engine, but i have a blocker: I don't know how to get the next steps of my current instance workflow. I want, from a running workflow, get the possible next steps, and choose one response action. I´m using waitFor, and i want, from a response, to choose which way to go
here is my code.

            builder
                .StartWith(context => ExecutionResult.Next())
                .WaitFor("event1", data => data.GUID)
                .Output(data => data.Object, step => step.EventData)
                .Then(context =>
                {
                    return ExecutionResult.Outcome((context.Workflow.Data as Data).Object as string);
                })
                .When(data => "yes").Do(then => then
                      .StartWith(context =>
                    {
                        Console.WriteLine("yes");
                        return ExecutionResult.Next();
                    }))
                .When(data => "no").Do(then => then
                      .StartWith(context =>
                      {
                          Console.WriteLine("no");
                          return ExecutionResult.Next();
                      })
                .Then(context =>
                {
                    Console.WriteLine("end");
                    return ExecutionResult.Next();
                });

Compensation Support

Related to the transaction support, there does not seem to be support to implement compensation mechanism in a workflow. Do you have any plans? How can it be implemented with existing primitives?

Workflow Designer

Do you have any plans to create a workflow designer?

Transaction support

As far as I can see, there does not seem to be support for transactions within a workflow. Do you have any plans? How can it be implemented with existing primitives?

parallel task

how do I do a parallel task?

something like a:
.parallell()
.do
.andDo

the only way i see to do a parallel task is with forloop or use [.when(true) for each parallel task]

please advise. tx

Workflow Tracking

I started an issue, but then I didn't follow up. Let me try again...

What is the best way to track long-running workflows, in terms of the state of the individual steps, etc.? At one point we had considered a separate table for persisting each step. I think that this would be a good idea since user-definable data could be persisted on a per-step basis. Ultimately, it would be nice to provide some kind of front end to all this (such as GoJS).

Daniel - do you have any plans for this?

Thanks again.

Access to ILoggerFactory in StepBody classes

Would it possible to added the ILoggerFactory instance to StepBody through IStepExecutionContext?

If it is possible with the existing code base, can you please extend one of the samples to leverage the same logger in StepBody derived classes?

Cannot use a boolean value for workflow data

I encounter an issue on mapping a boolean value in the data context to the output of a step.

For example, my data context is like below,

public class MyData
{
public string Name { get; set; }
public bool TrueOrFalse { get; set; }
}

I register a workflow as below.
host.RegisterWorkflow<MyWofkflow, MyData>();

Then, I build a step of my workflow as follow,

.WaitFor("MyEvent", (data, context) => context.Workflow.Id)
.Output(data => data.TrueOrFalse, step => step.EventData)

This throws an exception at WorkflowExecutor.ProcessOutputs. This is because the expression data.TrueOrFalse is not a MemberExpression since TrueOrFalse is a value type. The expression is actually resolved as an UnaryExpression.

I have a fix as below.

private void ProcessOutputs(WorkflowInstance workflow, WorkflowStep step, IStepBody body)
{
foreach (var output in step.Outputs)
{
var member = (output.Target.Body as MemberExpression);
var unary = output.Target.Body as UnaryExpression;
var resolvedValue = output.Source.Compile().DynamicInvoke(body);
var data = workflow.Data;

            if (member != null)
            {
                data.GetType().GetProperty(member.Member.Name).SetValue(data, resolvedValue);
            }
            else
            {
                if (unary != null)
                {
                    var memberExpression = unary.Operand as MemberExpression;
                    if (memberExpression != null)
                    {
                        data.GetType().GetProperty(memberExpression.Member.Name).SetValue(data, resolvedValue);
                    }
                }
            }
        }
    }

I think this fix the issue. I think the codes may have other areas which may encounter similar issue.

Is it possible to use this library in projects using .NET 4.7 but not using Core

this statement is not working in Asp.net MVC 5 (.NET 4.7)
IServiceCollection services = new ServiceCollection();\

Error I am getting is
The type or namespace name 'ServiceCollection' could not be found.

Is it possible to use this library in projects using .NET 4.7 but not using Core

Using BlockingCollection instead of busy loop in RunWorkFlows()

I have a question about RunWorkflows() in WorkflowThread.

In the worse case, a new workflow will be started after 500 milliseconds. Is it possible to use BlockingCollection(or other similar methods) to blocking each thread until getting a new workflow ID?

Question about JOIN workflow steps

Hi Daniel,

thank you for your awesome work! I have a question regarding implementing a "JOIN" workflow step which means a workflow step that waits until all predecessor steps are finished.

Another question is about an "END" step which terminates the current workflow. For example if there are multiple paths within a workflow and one path hits an END step this step should terminate the current WF.

Do have examples for such steps? Or could you at least describe how to implement this?

Regards,
Christian

GetWorkFlowInstances - MongoDb Persistance

When running the following code:
`
private readonly IPersistenceProvider _workflowStore;

    public WorkFlowMgmtController(IPersistenceProvider workflowStore)
    {
        _workflowStore = workflowStore;
    }

    // GET: api/values
    [HttpGet]
    public async Task<IActionResult> Get(WorkflowStatus? status, string type, DateTime? createdFrom, DateTime? createdTo, int skip, int take)
    {
            //var result = await _workflowStore.GetWorkflowInstances(status, type, createdFrom, createdTo, skip, take);
            var result = await _workflowStore.GetWorkflowInstances(null, "CreateAgencyWorkflow", null, null, 0, 5000);       
            return Json(result.ToList());
    }

`

I receive the following error:
Microsoft.AspNetCore.Server.Kestrel[13]
Connection id "0HL21G7BILODB": An unhandled exception was thrown by the application.
System.FormatException: An error occurred while deserializing the Data property of class WorkflowCore.Models.WorkflowInstance: Error resolving type specified in JSON 'MyClass'. Path '$type', line 1, position 119. ---> Newtonsoft.Json.JsonSerializationException: Error resolving type specified in JSON 'MyWorkFlowContext, MyWorkflow'. Path '$type', line 1, position 119. ---> Newtonsoft.Json.JsonSerializationException: Could not load assembly 'MyWorkFlow'.

If I remove the $ from $type, it seems to work - so not sure if it's a deserialization issue or not.

How to implement a cancellable waitfor

Hi

We have the requirement for a workflow, that we are waiting for two events in parallel. If one event is fired, we should finish the parallel processing and continue after the join. This means that if one event occurred, the other wait for should be cancelled.

I have seen that there is a cancellable step but I did not get it to work. Could anybody provide a solution on how to cancel a waitfor step?

Thank you in advance for your help!
Marc

Update: A parallel with a waitAny() whould also solve our problem.

Extend the OnError method to support custom retry strategies to handle the cloud transient errors

https://msdn.microsoft.com/en-us/library/microsoft.practices.transientfaulthandling.aspx

ExecutionResult.Persist runs the activity in an infinite loop.

This could well be a misunderstanding, but when I return ExecutionResult.Persist(context.PersistenceData); The activity that returns this just runs and re-runs indefinitely in a loop. I had expected this to suspend the workflow until a host.ResumeWorkflow is called. It is more than possible that I don't understand what ExecutionResult.Persist is supposed to do, but it feels like it is doing the wrong thing by re-running the activity in an infinite loop.

How to pass multiple values into a PublishEvent

Hi Daniel, I have a Step that is a WaitFor and I need to pass more than one value to the PublisEvent, how can I achive that?.

And example could be if you change the example 4 and in the MyDataClass add a second property

public string StrValue2 { get; set; }

then in the Step build you add another output

public void Build(IWorkflowBuilder<MyDataClass> builder)
        {
            builder
                .StartWith(context => ExecutionResult.Next())
                .WaitFor("MyEvent", (data, context) => context.Workflow.Id, data => DateTime.Now)
                    .Output(data => data.StrValue, step => step.EventData)
                   .Output(data => data.StrValue2, step => step.EventData)
                .Then<CustomMessage>()
                    .Input(step => step.Message, data => "The data from the event is " + data.StrValue)
                .Then(context => Console.WriteLine("workflow complete"));
        }

I try to define a new instance of MyDataClass and assign values into StrValue and StrValue2 and pass that object into the publishEvent
var values = new MyDataClass(); values.StrValue=1; values.StrValue2=2; host.PublishEvent("MyEvent", workflowId, values);

the problem is that the ProcessOutputs recive in the body the entity MyDataClass and dosn't now how to look for the values to assign in the output data.

Workflow example for .net framework 4.52

Is it possible to get an example for a "classic" .net framework implementation?

Best regards
Sigi

Question about the queue-ing mechanism and event handling

Hello,

We recently started working on a solution in which we are in need of a workflow framework, I came across your implementation and I would like to congratulate you on your effort to write such a framework and publish it here.
I have a few questions regarding it though:

• Is there any way to pass in some dynamic data as the key to the "WaitFor("eventName","eventKey")" function? If not are you planning on creating a way for this?
• When you are publishing a message, the host attempts to determine who the subscribers are and who the message is for, instead of just publishing the message, and let the subscribers just read the message if it is for them. Is there a way to make it the latter with some setting? If not, as above, are you planning on implementing such a feature?

Regards,
Attila

human workflow support ?

You have planned support for human workflow?

Multiple event subscription

Hello @danielgerlag,
Thanks all your hard work done and sharing it here. I want to know whether is it possible to subscribe a step to multiple events and provide a loopback as sample shown below.
builder .StartWith<CreateUser>() .Input(step => step.Email, data => data.Email) .Input(step => step.Password, data => data.Password) .Output(data => data.UserId, step => step.UserId) .Then<SendConfirmationEmail>(sendConfirmation => { sendConfirmation .WaitFor("confirmation", data => data.UserId) .Then<UpdateUser>() .Input(step => step.UserId, data => data.UserId) .EndWorkflow(); sendConfirmation.WaitFor("resendConfirmation", data => data.UserId).Then(sendConfirmation); });

Best Regards,
Nurhat

Using While with WaitFor

Hi Daniel,

Thanks for your great work on this. I am wanting to put a WaitFor Event in a While block and if a particular "Event" is received then Loop back to the beginning. Something like this.

.While(data => data.Event == "loop")
.Do(x => x
.StartWith()
.WaitFor("Event", data => "Key")
.Output(data => data.Event, step => step.EventData)
.Then()
.Then();

I have tried this and it appears that the Events don't get evaluated because the Workflow is locked. Is that expected, Can you advise an alternative workflow setup?

Thanks.
Aaron

How to deal with random collection of users?

I got questions about Users extension, for multiple users approval list:

1. We had workflow for multiple users approve , but those users could change. It seems while loop might work, but could you give some suggestion if there's better way?

2. Could it be possible to add user in above approval list (by position)?

3. in sample:
   host.PublishUserAction(openItems.First().Key, @"domain\john", answer).Wait();
   I need to get the "domain\john" string in UserTask/WithOption, how could it be done?

Thanks for your great work!

Process Event on Publish

Hi Daniel,

Is it possible to process an Event immediately rather than waiting for the RunnablePoller to run?
Any issues this may cause? There won't be a large number of events being published. Thanks.

Aaron

Asynchronous execution support

As I can see, you are only supporting synchronous workflow execution. It would be nice to have asynchronous alternatives.

Seperating Workflow Host and API

Hi Daniel,

Thinking of having a Workflow Host Farm with multiple Runnable Pollers, which does the hard work, and a single instance of API, which only accepts API calls - has not Runnable Pollers.

Trying to achieve this via below 2 approach. Do you reckon any quick way to achieve that?

1. API calls not blocked with Runnable Pollers retrieving steps from persistance storage and working on that. (Eg. Setting the Runnable Poller count to 0 on WorkflowOptions)

2. API does not need to be changed whenever a new workflow or step introduced.

For (# 2)

Having a quick look on code found out that Post method on WorkflowsController on WorkflowCore.WebAPI project requires to make main 2 calls(Registry.GetDefinition and WorkflowHost.StartWofklow). Both requires WorkflowHost has the registry list populated with workflow and step definitions.

(# 2) may be an overkill, but interested in your thoughts regarding both.

Async support to StepBody.Run

Hi Daniel,

Thanks for all the great work.

I am trying to implement steps with calls to async methods. Eg. HttpClient.GetAsync
Calling those methods on overridden StepBody.Run method generates an issue as the method requires to be marked as async. Currently achieving by defining async methods and calling the Result property of the result.

So, is it possible to run async methods inside StepBody.Run overridden method and wait for the result before evaluating the output/outcome or the next steps?

Concat multiple UserSteps

Hello Daniel,

how can multiple UserSteps be concatenated?

builder .StartWith(context => ExecutionResult.Next()) .UserStep("Do you approve A", data => "MYDOMAIN\\user", x => x.Name("Approval Step A")) .UserStep("Do you approve B", data => "MYDOMAIN\\user", x => x.Name("Approval Step B"))

does not do the trick. Calling "host.GetOpenUserActions(workflowId)" will only return one openItem.

Again, your help would be appreciated.

Regards,
René

Write documentation

The basic documentation needs to be written

Possibility to create workflow in runtime?

Are there such possibility? For example if I have external file with described workflow how I can create workflow in runtime?

Bug in GetWorkflowInstances

Hello Daniel. Failure to set a non-zero value to the take parameter in, for example, EntityFrameworkPersistenceProvider.GetWorkflowInstances fails at the line:

var rawResult = query.Skip(skip).Take(take).ToList();

and returns no result. Leads to an Internal Server Error (500) when called through WebApi.

Suggest that a non-zero value (i.e., 10?) be set if take parameter is zero.

Thanks for this great project.

Workflow steps history database

Would be nice if each workflow step is individually tracked and stored (i.e., date/time each step was completed with perhaps some user definable information also stored per completed step). Ultimately, this information could be reflected in an user interface such as GoJS.

If this is worth pursuing, I could start on this with some guidance from you.

State machine workflow

Do you have any sample for state machine workflow? I 'm using SharePoint workflow 2010 before and families with state machine flow.

Thanks,
Sup

Using in ASP.NET MVC 5....

Hi,

A bit of a noob question here, I am sure. I want to use this in an MVC application. Is there an example on how to get this going? Ultimately I am not looking to return content to the connected client but rather to kick off back end processes as a consequence of a workflow.

Any pointers to get this going would be great.

Nested Workflows

I just came across your library and it looks great! I am doing some research to see if it will fit my requirements.

This is a two part question:

1. Is it possible to nest workflows, so that one workflow starts another workflow and waits for it to finish with a result of some kind?

2. Is it possible to correlate a given workflow so that two workflows with a given correlation id cannot be started concurrently?

My use case is essentially a parent / child relationship with some long running processes.
I want to create a RegisterChildWorkflow and from within that workflow I want to kickoff the RegisterParentWorkflow, but there should only ever be one RegisterParentWorkflow running at a time. So that if I register two children they would both subscribe to the same RegisterParentWorkflow result.

Thanks!

Add new workflow to host dynamicaly

Hi Daniel. suppose I have a "Workflows folder" that contains workflow classes and I add them to the host by reflection and after that host will run then I add new workflow to the workflows folder when host is running. I was wondering if it's possible add new workflow at run time without stopping host when it is running.

Support "Microsoft.Extensions.DependencyInjection 2.0.0" package

This is nice and easy use WF framework. Do you have a plan to support .net core 2.0?

--
Thanks

How do I notify when a workflow instance is completed?

I see there is an event OnStepError in IWorkflowHost that notify the client when an error happens. However, I do not see any event thrown when a workflow instance is terminated/completed. Can we provide such an event? Or is there another way to notify the client?

Remove error log from WorkflowInstance object graph

A continuously repeating error could cause the error log to grow large quickly which would have a negative performance hit on reading workflows from the persistence store.

Joining workflow paths after multiple outcomes

Hello @danielgerlag

I'm just figuring out how to use workflow-core but I have problems joining workflow paths after branching the workflow due to multiple outcomes of a step. What I need is the following:

builder .StartWith<A>(x => x.Name("Switch")) .When(true) .Then<B>() .Then<C>() .End<A>("Switch") .When(false) .Then<D>() .End<A>("Switch") .Then<E>();

Unfortunately Step "E" is never reached.
How would I get A-B-C-E for A=true and A-D-E for A=false ?
I do not want to add step E to both paths to avoid redundancy.

Your help would be greatly appreciated.
Regards,
René

Handle exceptions in StepBody

Daniel - I'm not clear about exception handling in stepbodies. An exception thrown in a stepbody sets the workflow to status 0 and attempts re-tries continuously. The exception is logged to the console but not reflected back to the UI. Nor is there any record of the error kept in the database. What is best practice for handling an exception within the stepbody so that the user is alerted and the information can be conveyed back to technical support?

Corollary to this question: how does one response to an stepbody exception by terminating the workflow (and, hopefully, alerting the developers).

Thanks.

P.S. I'm spending a lot of time with this application now. Please direct me as to how I can help.

Document Approval Sample with GUI

Hi very nice work, wondering if there is a Web GUI planned.

Also, would be nice to see a document approval sample.

thanks

DeferSampleWorkflow can not completed

Launch Sample05, console show "Workflow started" and after 20 seconds it don't go to next step.

The context.PersistenceData always is null, so it can not fire Next()

Tests

Are there more detailed tests that you might be able to post yet?