datadog/extendeddaemonset

Kubernetes Extended Daemonset controller

extendeddaemonset's Issues

Unclear behaviour of selector field in ExtendedDaemonSet/ExtendedDaemonSetReplicaSet

Describe the bug
ExtendedDaemonSet has selector field: https://github.com/DataDog/extendeddaemonset/blob/main/api/v1alpha1/extendeddaemonset_types.go#L22 , which states:

	// A label query over pods that are managed by the daemon set.
	// Must match in order to be controlled.
	// If empty, defaulted to labels on Pod template.
	// More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors

This field is only used to initialize new ers:
https://github.com/DataDog/extendeddaemonset/blob/main/controllers/extendeddaemonset/controller.go#L528
In ers it is used to match with node (not pod) labels:
https://github.com/DataDog/extendeddaemonset/blob/main/controllers/extendeddaemonsetreplicaset/controller.go#L351
Not sure what the difference with specifying nodeSelector in spec.template.spec
It is also get printed in listing as "NODE SELECTOR":
https://github.com/DataDog/extendeddaemonset/blob/main/api/v1alpha1/extendeddaemonsetreplicaset_types.go#L118
Questions:

1. Does comment of this field needs updating? It states a label query over pods, being query over nodes instead?
2. What is the correct way to limit EDS pods to nodes with certain labels? (analog to spec.template.spec.nodeSelector in regular daemonset)

To Reproduce
Steps to reproduce the behavior:

1. Add node query to selector field in EDS
2. Pods are being scheduled on nodes using specified query

Expected behavior

1. Selector is used for querying pods
2. Nodes are queried using spec.template.spec.nodeSelector
3. spec.template.spec.nodeSelector get printed as "NODE SELECTOR" in ers listing

Environment (please complete the following information):

• Kubernetes version: 1.20.2
• Kubernetes distribution: vanilla bare metal

What is the status of this project?

Hello!

extendeddaemonset, or something similar, is a feature we've been wanting for our Kubernetes clusters. I see, however, that there hasn't been much activity on the project in the last year+. Is this project still in use internally at Datadog? Is it looking for new maintainers? What is the status of the 0.9.0 release?

Any information would help us make a more informed decision!

Thank you! This project looks great!

Option to disable automated promotion of canary

Is your feature request related to a problem? Please describe.
Sometimes some sophisticated automated/manual checks need to be run before validating canary, which are impractical to put in pod health checks.

Describe the solution you'd like
An option disableWaiting inside ExtendedDaemonSetSpecStrategyCanary with false default while, which disables automated validation of canary and DISABLE_WAITING environment variable in controller deployment which can be used to change default behaviour (to avoid the burden of putting disableWaiting inside every eds definition, if you want that all eds by default to be validated in manual mode).

Describe alternatives you've considered

1. Setting duration to a very big value (like math.MaxInt64)
2. Adding extendeddaemonset.datadoghq.com/canary-paused to every eds

Additional context

Feature request: Add `replicasPerNode` for reliable deployments of H/A node-local services

Hey!

Thanks a lot for sharing the great work and providing us the world's best observability service.

I'm filing this feature request believing this fits nicely within the scope of this project, while filling a big missing-piece in the current K8s ecosystem.

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

I've had a hard time designing and implementing robust and efficient deployments of node-local dns and all kinds of node-based ingress controllers(Nginx, envoy-based ones, etc.).

Basically the design issues distill into the below two points:

1. You should either use SO_REUSEPORT or K8s "Service Topology" (node-local dns) or externalTrafficPolicy: Local(ingresses) in order to expose a set of pods per node to the world with a minimum network hop.
2. Use of SO_REUSEPORT or externalTrafficPolicy: Local implies that you must have two or more pods "per node" in order to provide high-availability

Describe the solution you'd like

A clear and concise description of what you want to happen.

Add replicasPerNode to ExtendedDaemonSet.

For example, giving it a value of 2 results in an extended daemonset to produce at least 2 pods per node. I say "at least" as it can create one more pod per node in a canary deployment.

Implementation-wise, this would require:

• ReplicasPerNode *int added to the ExtendedDaemonSetSpec struct and the CRD updated accordingly
• The extended daemonset controller to be modified so that it can handle two or more non-canary extended-daemonset-replicaset
  • I believe it is currently "designed" to work with only one non-canary replicaset per extended daemonset. Although some functions like cleanupReplicaSet does seem like being designed to support arbitrary number of replicasets.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

A straightforward way to achieve point 2 is to deploy two or more daemonsets(I've been calling them "a set of daemonsets"). However, updating such set of daemonsets reliably isn't straightforward.

For example, kubectl apply -f set-of-daemonsets.yaml isn't usually what you want, because it can cause all the concurrent daemonset rolling update can start by bringing down all the daemonsets' pods in a specific node, which bring down the whole service provided by the pods on the node.

A possible solution would be to coordinate rolling-updates of the set of daemonsets. It's basically running kubectl apply -f first.daemonset.yaml and then kubectl apply -f second.daemonset.yaml and keep doing this until the last daemonset. With shell scripting, this can be a for loop. In a GitOps way, it can be a separate commit per daemonset.

The solution above works, but not in an ideal way. The ideal way would be a more declarative one. More concretely, a single kubectl apply or alike should be enough for a reliable update of a set of daemonsets.

Additional context

Add any other context or screenshots about the feature request here.

I'm eager to submit a pull request for this if this makes sense to be within this project.