jnpacker/work-placement

This controller uses the ManifetWork Spec and PlacementDecision CR's to

work-placement

This controller uses the ManifetWorkSpec and PlacementDecision CR's to create ManifestWork resources in specific clusters based on the status.decisions

Solves

Use a single resource for creating ManifestWork, across multiple clusters.

Security

The controller watches for PlaceManifestWork custom resources. When found, it reads the Spec.PlacementRef. If the placementRef has a valid PlacementDecision resource available, a ManifestWork will be created in each cluster listed in the PlacementDecision.Status.Decisions.

This relies on ManagedClusterSet to restrict the PlacementDecision to only authorized clusters. Since normal users do not have create, update or patch for PlacementDecision, they can not effect the decision outcome.

The PlaceManfiestWork.Spec.PlacementRef is namespace scoped, so you can not use a PlacementDecision that is not in the same namespace. This also controls access, as a user can only reference Placement's that are in the same namespace where they have permission to create the PlaceManifestWork resource.

For Placement to create a PlacementDecision, it must be bound to a ManagedClusterSet via a namespace binding. The user will not be able to target clusters that their Placement resource does not have permission to access.

How it works

The CRD is simple and draws from the ManifestWorkSpec. It's Spec contains a reference to the ManifetWorkSpec, and an object reference to put a Placement resource for cluster targeting. The Placement resource name is referenced as a label on its PlacementDecision which is how the controller is able to look up the PlacementDecision resource.

The controller updates the conditions for the PlaceManifestWork tracking whether it has found the PlacementDecision to use for distribution of the ManifestWorkSpec and how successful it was creating or updating ManifestWorks

The controller will reconcile the PlaceManifestWork against all the existing ManifestWork each time it reconciles. If no change occured to the PlaceManifestWork resource, the reconcile happens every 120s. When reconciling, first the PlacementDecisions.status.decisions is reconciled. The ManifestWorkSpec is updated or added to all clusters in the PlacementDecision (will loop through multi-page decisions). Once all chosen clusters have had their ManifestWork updated or created, it will delete any ManifestWork that are no longer part of the PlacementDecision status.

Laptop usage

Build it locally

make build

Apply CRD

make install

Run it locally in your dev environment

make run

Remove CRD

make uninstall

Kube controller deployment

Build image

Contributing...

export REPO=registry.com/repo-name/ # It is important to include the trailing slash
make docker-build
make docker-push

Deploy the controller

make deploy

Undeploy the controller

make undeploy

Sample PlaceManifestWork

This sample assumes you have Open-Cluster-Management.io setup

• You will need at least 1 ManagedCluster to be able to leverage the PlaceManifestWork

• Creating a ManagedClusterSetBinding

  kubectl apply -f samples/clustersetbinding.yaml

• Creates a placement resource in the default namespace targeting the all-clusters clusterSet

  kubectl -n default apply -f samples/placement.yaml

• Creates the PlaceManifest work in the default namespace

  kubectl -n default apply -f samples/place-manifestwork.yaml

Enhnacements

• See the Issues tab

work-placement's Issues

Use a unique 5 character value in name from generated ManifestWork resources?

Problem

Do we need to make the generated ManifestWorks name unique?

Proposal

Generate a unique 5 character suffix that is appended with a dash.

When reconciling the ManifestWork do not stop on error

Problem

The reconcile loop currently stops processing when it hits a problem with create, update or delete.

Proposal

• We will loop through all create and updates, but log.Error if there is a problem. Also set the ManfiestWorkApplied condition to False with reason PartiallyProcessed
• For a reconcile once finished after 30s.

Wake up the PlaceManifestWork controller to reconcile when PlacementDecision changes

Problem

If the PlacementDecision.Status.Decisions changes, we should reconcile the ManifestWorks that are being targeted.

Solution

Add a watch for PlacementDecision. Possibly put some logic to check for a label on the Placement or PlacementDecision that makes for a nice filter to connect PlaceManifestWork and PlacementDecision

Looking for comments.

Make reconcile loop wait time configurable for the controller

Problem

Default reconcile if no change is made to PlaceManifestWork, is 120s. This means that if you change one of the child ManifestWorks it will be reconciled back to the ManifestWorkSpec that was found in PlaceManifestWork.

Solution

Instead of making it an environment variable, default to 120s and read a configMap to get any custom timing settings. This allows other options to be enabled.

Looking for comments here.