refactoring the YAMS model: from deployments to clusters and node roles

[microsoft]

refactoring the YAMS model: from deployments to clusters and node roles

[jdom]

All good ideas. I would also agree with @nehmebilal on doing small changes at a time, as it's easier to look at what's possible once we are closer to the goals.
In my opinion, the node type addition can already enable a lot. If we do that, I would still consider deprecating the DeploymentId name in favor of something like YamsClusterId, to disambiguate with whatever hosting technology Yams runs on. Node type on the other hand doesn't need a Yams prefix in it, as it is a logical name that only applies inside the (Yams) cluster, and it doesn't matter if other hosting technologies reuse the same name

[nehmebilal]

Here is what I am thinking to make it possible for users to add additional properties such as "NodeType" easily.

The DeploymentConfig.json will look as follows:

{
    "Applications":
    [
        {
            "Id": "app1",
            "Version": "1.0.0",
            "YamsClusters": [ "YAMS_CLUSTER_ID" ],
            "Properties":
            {
                "NodeType": "PROD"
            }
        },  
        {
            "Id": "app1",
            "Version": "1.0.1",
            "YamsClusters": [ "YAMS_CLUSTER_ID"],
            "Properties":
            {
                "NodeType": "QA"
            }
        },
    ]
}

Yams will require an implementation of the following interface to match the deployments to clusters:

public interface IAppDeploymentMatcher
{
    bool MatchesCurrentCluster(AppDeploymentConfig appDeploymentConfig);
}

The default implementation uses the Yams cluster id:

public class YamsClusterIdDeploymentMatcher : IAppDeploymentMatcher
{
    private string _yamsClusterId;

    public ClusterIdDeploymentMatcher(string yamsClusterId)
    {
            _yamsClusterId = yamsClusterId;
    }

    public bool MatchesCurrentCluster(AppDeploymentConfig appDeploymentConfig)
    {
        return appDeploymentConfig.YamsClusterId == _yamsClusterId;
    }
}

To support NodeType, the user can supply a custom implementation as follows:

public class CustomAppDeploymentMatcher : IAppDeploymentMatcher
{
    ...

    public bool MatchesCurrentCluster(AppDeploymentConfig appDeploymentConfig)
    {
        if(appDeploymentConfig.YamsClusterId != _yamsClusterId)
        {
            return false;
        }

        // Verify that the node type matches
        string nodeType = "";
        if(appDeploymentConfig.Properties.TryGetValue("NodeType", out nodeType))
        {
            return nodeType == _nodeType;
        }
        return false;
    }
}

You could even write a matcher that matches on a start date for a scheduled job. Another example would be a matcher that makes a service call to fetch some configs and decide if the app should be deployed or not.

How does that sound?

[nehmebilal]

Just to list some of the issues discussed here:

• I would like to have a NodeType in my DeploymentConfig.json so that I can separate my deployments per type into different clusters. This feature would also be useful for Orleans where backend and frontend apps are deployed to separate WorkerRole's. I personally think we should generalize this and make it possible to add any number of user meta-data to the DeploymentConfig.json. I have some ideas but I'll go in more details once we have a dedicated issue :)
• I would like to have separate DeploymentConfig.json for unrelated apps. This is totally doable today. For the SQL scenario, you can pass to your host exe the key of the row where the corresponding DeploymentConfig.json is stored. Then, pass this down to your implementation of IDeploymentRepository which will fetch the right data. It's also doable using the default blob storage solution by using separate storage accounts for unrelated apps. By the way, you can even store the json in different format and then populate the DeploymentConfig class using the AddApplication Apis.

I'm sure there's more between the lines :)

[danvanderboom]

Having a ClusterId and NodeTypeId, and no longer use DeploymentId, would by itself be a large enabler of other things. The SQL script I shared would become possible, for example.

The config representations can be done now, in one way or another, and one form can be easily mapped to another. So yeah, it's really that use of Cluster & NodeType abstractions that are the thing to focus on.

[nehmebilal]

@danvanderboom Please take a look at #34 and let us know if that works for you.

[danvanderboom]

Great work!

It might take me a week or two before I get a chance to test it out and upgrade our current project to use the newest version, but I'll report back when I do.

[nehmebilal]

There are several ideas here but I'm going to focus on the NodeType proposal to keep the discussion focussed. If I understand correctly, @danvanderboom is proposing adding a NodeType (let's call it that for now) field to the DeploymentConfig.json as follows:

{
    "Applications":
    [
        {
            "Id": "app1",
            "Version": "1.0.0",
            "DeploymentIds": [ "YAMS_CLUSTER_ID" ],
            "NodeType": "PROD"
        },  
        {
            "Id": "app1",
            "Version": "1.0.1",
            "DeploymentIds": [ "YAMS_CLUSTER_ID"]
            "NodeType": "QA"
        },
    ]
}

The Yams host on each node will know its type and will pull the appropriate deployments based on their types as described in the json above.

The current solution is to concatenate PROD or QA to the DeploymentId (which should be renamed to YamsClusterId).

@danvanderboom Is that accurate?

[danvanderboom]

Here's how I see it looking, ideally, for a system with three node types in a QA cluster, and where each node type has two services configured to be installed.

{
  "ClusterId": "MyProduct-QA",
  "NodeTypes": [
    {
      "NodeTypeId": "Storage",
      "Services": [
        {
          "Id": "StorageService",
          "Version": "3.0.0.0",
          "Executable": "<path to .exe>",
          "ExecutableArgs": "<arguments with tokens for special values>",
          "Properties": [
            { "ConfigFilesPackageUrl": "<URL to download config files package>" },
            { "AnotherCustomProperty": "..." }
          ]
        },
        {
          "Id": "LoggingService",
          "Version": "1.0.0.0",
          "Executable": "<path to .exe>",
          "ExecutableArgs": "<arguments with tokens for special values>"
        }
      ],
      "Properties": [
        { "SomeSpecialService": { "Id": "LoggingService", "Version": "1.0.0.0" } }
      ]
    },
    {
      "NodeTypeId": "ComputeA",
      "Services": [
        {
          "Id": "ComputeServiceA",
          "Version": "3.0.0.0",
          "Executable": "<path to .exe>",
          "ExecutableArgs": "<arguments with tokens for special values>"
        },
        {
          "Id": "LoggingService",
          "Version": "1.0.0.0",
          "Executable": "<path to .exe>",
          "ExecutableArgs": "<arguments with tokens for special values>"
        }
      ]
    },
    {
      "NodeTypeId": "ComputeB",
      "Services": [
        {
          "Id": "ComputeServiceB",
          "Version": "1.0.0.0",
          "Executable": "<path to .exe>",
          "ExecutableArgs": "<arguments with tokens for special values>"
        },
        {
          "Id": "LoggingService",
          "Version": "1.0.0.0",
          "Executable": "<path to .exe>",
          "ExecutableArgs": "<arguments with tokens for special values>"
        }
      ]
    }
  ]
}

Then you'd keep separate configurations for the other environments. In my case where I'm storing these in SQL Server, each cluster's config/manifest is a separate row in a table.

[danvanderboom]

So when YAMS starts up, it would need to know two things:

1. ClusterId
2. NodeTypeId

[danvanderboom]

This is a cluster manifest. And we'd have separate manifests, one for each cluster/environment.

This can be used in a file based way, versioning with code, differentiating its details by solution configuration, and loading the .json file to configure YAMS... and it's also friendly for storing as rows in a database table in case you want centralized configuration.

[danvanderboom]

With a Properties dictionary at the Cluster level as well, these settings could be made available to YAMS extensions like deployment storage providers. This would centralize and make convenient configuration for any and all YAMS extensions, as well as any switches or settings that may alter the operation of YAMS (not sure if there are any of these).

[danvanderboom]

If we could call:

IDeploymentRepository.FetchNodeTypeConfig("MyProduct-QA", "ComputeA")

... we'd get back a NodeTypeConfig or NodeTypeManifest object, which would be one of these:

{
  "NodeTypeId": "ComputeA",
  "Services": [
    {
      "Id": "ComputeServiceA",
      "Version": "3.0.0.0",
      "Executable": "<path to .exe>",
      "ExecutableArgs": "<arguments with tokens for special values>"
    },
    {
      "Id": "LoggingService",
      "Version": "1.0.0.0",
      "Executable": "<path to .exe>",
      "ExecutableArgs": "<arguments with tokens for special values>"
    }
  ]
}

And to not be totally confusing, "Services" are normally called "Applications" in YAMS.

A call to:

IDeploymentRepository.FetchClusterConfig("MyProduct-QA")

... would give us the whole big cluster manifest JSON fragment I shared previously.

[danvanderboom]

A SQL Server database setup script for supporting this kind of model:
https://gist.github.com/danvanderboom/a584bbf2fbdb4a626bb72912bd4828f5

[nehmebilal]

Thanks for the ideas @danvanderboom. I doubt that we will make such major changes in one go. It would be great if we can focus on one issue at a time. You never know, It's possible that there are existing solutions to some of the challenges you're trying to address.

My suggestion would be to identify functionality that is not supported (or can be improved) and open a corresponding focused issue to discuss each. Baby steps :)

[jdom]

I think that's reasonable. After this we can update the Cloud Services sample to use a node type that matches the worker role name (in addition to the deployment id) if it supports multiple node types.
I assume these values can also be used as tokens in the command line arguments when launching the external application?

[danvanderboom]

That will work for me, @nehmebilal. Thanks for taking the time to focus on this!

[nehmebilal]

@danvanderboom No problem :)

@jdom Good point. Yes, we should be able to make the properties available as tokens that can be used in the AppConfig.json.

[nehmebilal]

1.1.0-rc is up!

[nehmebilal]

Closing.. Feel free to re-open if you find any issues.