microsoft/forge

A Generic Low-Code Framework Built on a Config-Driven Tree Walker

Microsoft.Forge.TreeWalker NuGet link

Please use the Issues tab for questions, feature requests, bug reports, etc.. Thank you!

Check out the Forge QuickStart Guide to get hands-on experience running a complete Forge implementation locally in under 60 seconds!

4/15/2022: Check out the Beta release of the ForgeEditor Web version! https://forge-editor.microsoft.com/ or https://aka.ms/forge-editor.

What is Forge?

Forge is a generic low-code framework built on a config-driven tree walker. Forge is a great library for developers wanting to build and run their own low-code workflow engine from inside their application.

Similarities to other low-code solutions:

• Walks a decision-tree-like structure that is defined in a JSON config file given by your application.
• Executes actions and can use the results to decide which child node to visit next.
• Edit config files through a UI experience.

Key Differentiators:

• Runs directly in your application! No need to rely on outside services to run your workflows. Forge is a light-weight library that is easy to import and integrate into your application.

• Define your own Actions. Forge makes it easy to write simple C# classes that can be executed as Actions from the tree.

• Give Forge scoped access into your application. Expose data models, internal classes, service clients, etc. to Forge and then access them directly from the tree.

• Roslyn integration allows you to replace hardcoded JSON property values in the tree with C# code-snippets. Forge dynamically executes these at runtime to fetch realtime data. For example, fetch data from your application to give as input to an Action.

• Calls back to your application with objects from the tree. Forge dynamically creates objects defined on the tree and passes them to callbacks and Actions. Callbacks before and after visiting each node provide more places to plug in code.

• Built-in persistence interface allows tree walker sessions to rehydrate without missing a beat.

• Instantiate sessions with different state from your application depending on the trigger.

• Facilitates ecosystem building with clearly differentiated contributors: application owner, tree authors, and action authors. Application owners can use Forge to build their own workflow framework for their service. Different teams can then use their framework to author actions and trees.

Forge components at a high-level

Forge has 3 major components: ForgeTree, TreeWalker, and ForgeEditor.

• ForgeTree is the JSON data contract that defines the tree structure. It contains normal tree-concept objects such as TreeNodes and ChildSelectors, as well as TreeActions and other properties.

  In this example, the Container_TreeNode is an Action type node. It executes a CollectDiagnosticsAction with an Input object containing a Command property. The value of Command is set through a Roslyn expression that calls into the application to get the UserContext.GetCommand() result. In the child selector we see another Roslyn expression that gets the persisted action response, and visits the Tardigrade_TreeNode if the Status is successful.

"Container_TreeNode": {
    "Type": "Action",
    "Actions": {
        "Container_TreeAction": {
            "Action": "CollectDiagnosticsAction",
            "Input": {
                "Command": "C#|UserContext.GetCommand()"
            }
        }
    },
    "ChildSelector": [
        {
            "ShouldSelect": "C#|Session.GetLastActionResponse().Status == \"Success\"",
            "Child": "Tardigrade_TreeNode"
        }
    ]
}

• TreeWalker takes in the ForgeTree and other parameters, and walks the tree to completion. It calls application-defined callbacks and actions, passing in dynamically evaluated properties from the ForgeTree. The TreeWalker makes decisions at runtime about the path it walks by utilizing Roslyn to evaluate C# code-snippets from the ForgeTree.

• ForgeEditor Web version is available at https://forge-editor.microsoft.com/ or https://aka.ms/forge-editor. ForgeEditor is an Electron application that allows you to visualize and edit the ForgeTree in a clean UI experience. It contains features such as: tree visualization, buttons to create/delete TreeNodes, auto-complete when editing JSON file, text highlighting when hovering over TreeNode, etc..

Advantages of using Forge

• Clarity: Allows users to intuitively walk a visualized tree and understand the workflow logic.
• Versatility: Once you understand the tree, you can easily add or update nodes, actions, child selectors, paths, etc..
• Extensibility: The dynamic objects and properties allow for new functionality to be seamlessly piped into the tree model. For example, adding rate limits to the BeforeVisitNode callback.
• Velocity: Updating and deploying ForgeTree schema files is much quicker/easier than deploying code. Allows for hot-pushing updates while your application is running. New behaviors that would take weeks to add to legacy codebases turns into minutes/hours with Forge.
• Debuggability: Add logging to the built-in callbacks before and after visiting each node and inside Actions. This allows users to view statistics on their historical data. Statistics can also be viewed in ForgeEditor as a heatmap to highlight "hot" nodes/paths (internal application only).
• Model Verification: It can be easy to typo an Action name or other property in the ForgeTree since they are JSON strings. To help with that, Forge includes the ForgeSchemaValidationRules.json file containing JSON schema rules for ForgeTrees. These rules can be augmented to contain application-specific properties/values, such as enforcing a set of known ForgeActions and ActionInputs. These rules are used in ForgeEditor to help auto-complete properties and highlight typos (internal application only).
• Stateful (optional): The IForgeDictionary is used to cache state while Forge walks the tree. This can be implemented by the application to be stateful if desired, but a Dictionary implementation is built-in. This allows applications to pick up where they left off after a failover.

User Story: Azure-Compute Repair Manager

Microsoft Azure-Compute's repair manager service is backed by Forge. Fault information is passed to the TreeWalker and a ForgeTree schema is walked. Depending on the type of fault and a handful of other checks, TreeWalker may attempt in-place recovery actions. If these actions were unsuccessful in mitigating the issue, diagnostics are collected and Tardigrade may be performed. You can read more about Azure’s recovery workflow in this blogpost: https://azure.microsoft.com/en-us/blog/improving-azure-virtual-machines-resiliency-with-project-tardigrade/

"The biggest (and unexpected) benefit of using Forge has been the democratization of contributors and the velocity of pushing updates. It used to take weeks to implement new features on the old service, and very few engineers had the expertise to make such changes. Since moving our repair workflows to Forge, new features/behaviors/paths are often added by teams that request them. This is possible because of the clarity that visualizing the tree brings. After initial ramp-up time, new contributors are then able to send code reviews out for changes within a day. They also basically become experts themselves and help onboard their teammates. It's magical! A cross-team ecosystem has been organically created and folks are eager to contribute."

Further Reading

Check out the Wiki pages for a deeper dive into Forge, as well as How-To guides!

For more high-level overview details, check out the ForgeOverview page.

Ready to get hands-on experience with Forge? Check out the Forge QuickStart Guide page.

Ready to dig a bit deeper? Check out the "How To:" guides for in-depth details for each group of contributors:

• Application owner - How To: Use Forge in my Application and How To: Extend Forge in my Application
• ForgeTree author - How To: Author a ForgeTree
• ForgeAction author - How To: Author a ForgeAction

Contributing

Interested in contributing to Forge? Check out the Contributing page for details.

License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT license.

forge's Issues

FinallyTrees - Ability to execute code before ending the tree walker session

This needs a lot more thought, but here's the rough idea. Ability to define some code that will get executed before ending the tree walker session.

The code/Actions(?)/subroutines that get executed could be a growing list as the tree is walked.
Or perhaps they could be added to the ForgeAction definitions themselves.

This would be useful to ensure clean-up code gets executed, even on failure.

Add callback after LeafNode ActionResponse gets persisted

This gives users a chance to log the LeafNodeSummaryAction, for example. This is especially useful in Subroutines since the app isn't walking the session so has no access.

Re-execute Node even when there is a persisted response

Hi All,

We have an use case where we execute a node and if none of the child selectors match, we want to retry the same node after few min to see if any of the child selectors match. But the current behavior is in a way that if a persistent response is found the execution of the node is skipped.

How can I have the node to be re-executed even when the persisted response is found?

This is also helpful when we have loops in the forge schema

Make RetryExhaustedOnAction and TimeoutOnAction strings accessible from Tree

Instead of hard-coding "RetryExhaustedOnAction" and "TimeoutOnAction", let's add public const strings or enums so they are more accessible. They should be accessible through the tree/schema so users can easily check the ActionResponse.Status.

Possible solution is to add a new class to the parameters of the Roslyn script.

"ShouldSelect": "C#|(await Session.GetLastActionResponseAsync()).Status == FooEnums.TimeoutOnAction",

Allow TreeWalkerSession to call WalkTree multiple times

Today a TreeWalkerSession can only call WalkTree one time. This is because the internal CancellationTokenSource (walkTreeCts) is created in the constructor, and cancelled at the end of WalkTree to make sure all tasks get cleaned up.

However, there is a common pattern to intentionally halt the tree walker by throwing an exception. The application then retries walking the tree at a later time depending on the scenario.

• E.g. Halt tree walker in BeforeVisitNode due to hitting a rate limit, then retry walking the tree in 1 minute.
• E.g. Halt tree walker to wait for an event, then walk the tree again.

The issue is because TreeWalkerSession can only call WalkTree once, the application must initialize a new TreeWalkerSession for every retry attempt. Even though all the same parameters are often used. Being able to use the original TreeWalkerSession would simplify and optimize the multi-WalkTree scenarios.

Solution idea:

• Recreate the walkTreeCts at the top of WalkTree. This would allow WalkTree to be called multiple times.
• Before recreating walkTreeCts in WalkTree, consider if we should cancel and dispose the existing walkTreeCts.
• We also need to keep the initialize walkTreeCts in the constructor since some applications call VisitNode instead of WalkTree, and we don't want to break them.
• Unrelated, but we should also call walkTreeCts.Dispose() after calling Cancel() in the CancelWalkTree() method.

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

CachedVariable for ChildSelectors

• Add CachedVariable (name pending) property that caches the result of an expression for use in the ChildSelectors.
	○ A common pattern in ChildSelectors is a switch statement, where you calculate some value and visit different children depending on the result. The issue today is this calculation must be done for each child selector. This feature would allow you to calculate the result once and use it directly multiple times. 
	○ Allows you to evaluate an expression and save the result (non-persisted?).
	○ ChildSelectors then have access to that variable to simplify their ShouldSelect statements.
	○ Consider allowing the variable to be a dynamic object. So could be bool, string, object, etc.. This allows users to set as many properties as they wish.
	○ Consider what using a dynamic object would look like. If we have to wrap it every time with (Session.CachedVariable as string), then the usability goes down. In this case, it may be better to just set it as a string. Users could still use it as a bool or object by converting the string if they desire.
	○ Consider if this should be localized to just the current TreeNode's ChildSelector, or if you can access parent's CachedVariables as well.
	○ Consider if this should be tied to ChildSelector at all. Perhaps just adding a way to persist data in the ForgeStateDictionary from the tree would be cool.
		§ Perhaps they both could happen though. This would be a helper property to make the feature intention clear. If we decide to add more functionality later for setting ForgeState, we can still do that.
	○ Considerations:
		§ Persisted vs non-persisted
		§ Useable from ChildSelector only or everywhere?
		§ Save as String vs Object
		§ Cache result vs expression itself. Specify if you want to cache the expression result, or reevaluate.
		§ Consider defining at the tree level.
		§ Add flag to dictate behavior of CachedVariable. Whether or not to reevaluate each time. Could be unclear what the behavior is by default, make sure to set up the right expectations.
	○ Issue with tree structure
		§ How to enforce that people are calling it at the right time? For example, the expression could GetOutput of a TreeNode that hasn't executed yet.
	○ Issue with node structure
		§ If we decide to persist, not having good discoverability on parent nodes persisted data.
	○ Example Old way:
                "ChildSelector": [
                    {
                        "ShouldSelect": "C#|(await Session.GetLastActionResponseAsync()).Status == \"Failure\"",
                        "Child": "SoC_HS_RebootOnly_Failure"
                    },
                    {
                        "ShouldSelect": "C#|(await Session.GetLastActionResponseAsync()).Status == \"Timeout\"",
                        "Child": "SoC_HS_RebootOnly_Timeout"
                    },
                    {
                        "ShouldSelect": "C#|(await Session.GetLastActionResponseAsync()).Status == \"Success\"",
                        "Child": "SoC_HS_RebootOnly_Success"
                    }
	○ Example New way:
                "ChildSelectorVariable": "C#|(await Session.GetLastActionResponseAsync()).Status",
                "ChildSelector": [
                    {
                        "ShouldSelect": "C#|Session.CachedVariable == \"Failure\"",
                        "Child": "SoC_HS_RebootOnly_Failure"
                    },
                    {
                        "ShouldSelect": "C#|Session.CachedVariable == \"Timeout\"",
                        "Child": "SoC_HS_RebootOnly_Timeout"
                    },
                    {
                        "ShouldSelect": "C#|Session.CachedVariable == \"Success\"",
                        "Child": "SoC_HS_RebootOnly_Success"
                }

Feature Ask - Allow application to choose when to invoke the forge action retry policy

Today the Action retry policy kicks in if the action throws an exception. In many cases, the application wants to retry the action based on the StatusCode/Output of the action, not just on exception. In my case, I want to retry the action "n" if it returns a specific status code.

TreeInputType feature request

Problem: There is a contract missing between Subroutine author and SubroutineAction user, which is the TreeInput type. Because the TreeInput type of the SubroutineAction is a dynamic object, the TreeInput property can be defined in various ways. This provides great flexibility, but makes it difficult for the author and user to trust what the expected properties should be.

Feature Suggestion: Add some way to define the expected TreeInput properties inside each individual Subroutine. Forge would then create the specified TreeInput object (either known or dynamic) when calling the SubroutineAction.

Considerations:

• It would be cool if adding a Subroutine TreeInput definition would not require any app code changes. i.e. A config-only change.
• Ideally, known types could also be used in TreeInput. If the application already knows of this type, then it would not require an app deployment. But if it is a new type, then an app deployment would be necessary, which is fine.
• If a TreeInputType is defined as a dynamic object on the tree, then it would be expected that Forge create a dynamic object. Similar to how Forge creates the TreeInput or the Properties property today.
• If the TreeInputType is defined as a knownType however, Forge would be expected to create that type. Similar to how Forge creates an InputType for ForgeActions.

EvaluateDynamicProperty inconsistent behavior for version 1.0.55 [Bug]

I bumped up Forge version from 1.0.52 to 1.0.55 and I'm getting EvaluateDynamicProperty failed to parse schemaObj errors that was not happening before.

Error

EvaluateDynamicProperty failed to parse schemaObj: {
  "UserParameters": "C#|UserContext.GetUserParameters()"
}, knownType: Microsoft.Azure.Solutions.TroubleshooterV2Library.WorkflowEngine.Forge.ForgeActions.DataInput.

Action Input within Schema:

"Input": {
          "UserParameters": "C#|UserContext.GetUserParameters()"
}

Input Data Model

public class DataInput
{
      public Dictionary<string, string> UserParameters { get; set; }
}

User Context which has the function GetUserParameters()

public Dictionary<string, string> GetUserParameters()
 {
            return this.userParameters;
 }

I am sure that I have my Forge set up correct because everything worked as expected in version 1.0.52. Has something changed with EvaluateDynamicProperty 1.0.55 that I do not know? Or is this a bug with 1.0.55

What is the current status of Forge and ForgeEditor?

Can you please explain the current status of Forge within Microsoft? It looks an interesting project (internal tool). However of limited utility without the ForgeEditor graphical editor tool.

The apparent promise in README.md to open ForgeEditor by the end of 2019 became "coming soon" in Jan 2020. We are now nearing the end of 2021 and it hasn't appeared.

Does this imply Forge is no longer actively positioned for use outside MS?

How to download/use ForgeEditor?

Hi,

I'm starting to learn Forge to use its DAG feature for my team. I am struggling with finding working examples and have forge editor. I see the beautiful image out of json in the first page summary of this repo, but I couldn't have it downloaded and working on my box.
I'm using 'Forge.TreeWalker - 1.0.29' nuget package for my learning.

Any help would be really appreciated.
TIA

Feature Ask - Allow application to run custom code when Forge sees exceptions

It would be great if Forge allowed applications to provide a delegate that Forge would invoke, each time any exception is thrown and caught by Forge. That way the application can alter state/prepare of the exception. In my case, if the machine is low on memory (causing OutOfMemoryException), I want to kill the process and don't want Forge to finish walking the tree, so that it can resume walking on a new machine.

SubroutineInput - StartOverOnRetry flag

• Current behavior is that Forge will remember the CurrentTreeNode of the Subroutine and visit that node on failover/retry.
• What if the user would like to start from the RootTreeNode on retry?
• I could see either behavior making sense, so we should add a property flag in SubroutineInput to determine the desired behavior.
• Probably would need to delete all the state from subroutine session before retrying. I don't want to parse through the data though, so perhaps just starting a brand new subroutine SessionId would suffice.

Can we throw custom exception from child in schema?

I have an use case where if a specific child selector becomes true, I want to throw an exception. is there a way to do this today?

Does Forge expose any API to validate schemas

I see that in unit test proj, contracts\ForgeSchemaValidationRules.json is used to validate the schema, similarly is there a public API that users can make use of to validate schemas?

Can Forge re-walk previous nodes?

Forge has the GetCurrentTreeNode which fetches the current node. Is there a 'SetCurrentTreeNode' which will set the current node to one of the previous nodes?

Ex) Lets say that a tree has (1),(2),(3),(4),(5) nodes with schema:
(1)->(2)->(3)->(4)->(5)

The current node is (4) and Forge dictionary stores data on the decisions made on (1),(2),(3).
(1)->(2)->(3)->(4)

At this point, we decide we want to rewalk the tree from node (2), where Forge dictionary will remove any data written from nodes after (2)
(1)->(2)

Is behavior like this possible?

Does Forge have functionality to pause and continue workflow?

An example scenario would be Forge waiting for an User input to continue the workflow.

Add option to construct a TreeWalkerParameters with a ForgeTree instead of JsonSchema

Adding option to construct a TreeWalkerParameters with a ForgeTree object directly, instead of the Json string. This saves the TreeWalkerSession from having to deserialize the json.

To take full advantage of this small feature, it is suggested for your application to cache the deserialized ForgeTree object(s), and use those when constructing TreeWalkerParameters.

We were seeing issues where the deserialized ForgeTree objects were ending up on the LargeObjectHeap for large schemas. These objects do not get garbage collected as efficiently as objects on the stack, and were leading to memory fragmentation.

Casting ActionResponse Output object to custom types in Forge schema

I am trying to send a dictionary as output in ActionResponse and here are two examples

1. new ActionResponse { StatusCode = 0, Status = false.ToString(), Output = JsonConvert.SerializeObject(outputDict) }

Forge : "ShouldSelect": "C#|UserContext.DeSerializeStringToObject((string)Session.GetLastActionResponse().Output)["nssLifeCycleCheck"] == "False""

public Dictionary<string, string> DeSerializeStringToObject(string output)
{
return JsonConvert.DeserializeObject<Dictionary<string, string>>(output);
}

This is working fine

2. new ActionResponse { StatusCode = 0, Status = false.ToString(), Output = JsonConvert.SerializeObject(outputDict) }

Forge : "ShouldSelect": "C#|UserContext.DeSerializeStringToObject< Dictionary<string, string>>(>((string)Session.GetLastActionResponse().Output)["nssLifeCycleCheck"] == "False""

public T DeSerializeStringToObject(string output)
{
return JsonConvert.DeserializeObject(output);
}

This is not working.

The only difference I see between the two examples is that Instead of using generic Deserialization, In the first example I am doing specific de-serialization to Dictionary<string, string> type. What am I missing here.

Also, Please let me know if there is a way I can send Dictionary directly in output and use it on Forge, I tried the below and getting exception.

"ShouldSelect": "C#|((JObject)Session.GetLastActionResponse().Output).ToObject<Dictionary<string, string>>()["nssLifeCycleCheck"] == "true"",

"ShouldSelect": "C#|((Dictionary<string, string>)Session.GetLastActionResponse().Output)["nssLifeCycleCheck"] == "False"",

Migrate to NetStandard 2.0

This library is .NET framework 4.6.2, is there a plan to migrate to NetStandard 2.0, convenient for use on .NET Core 3.1

Shipping symbols for Forge libraries

Today in Forge nuget package, we don't have symbols for the dll's and exe's. We are also not able to look them up in public symbol servers. Having symbols would help us in debugging Forge libraries as well.