Add daemonset node tolerations support

[storageos]

Add daemonset node tolerations support

[Arau]

Hi @tobiaskohlbau,

that makes complete sense. We are adding this into the roadmap. StorageOS cluster-operator is under heavy development to add more functionality and support, if for the moment and until we add the tolerations, you want to be able to edit the DaemonSet, you can use a different method to install, https://github.com/storageos/deploy/tree/master/k8s/deploy-storageos/standard. That method lets you edit the manifests, for instance https://github.com/storageos/deploy/blob/master/k8s/deploy-storageos/standard/manifests/040_daemonset.yaml_template.

Any PR would be highly appreciate it and we would review it accordingly!

[tobiaskohlbau]

Hello @Arau I've started implementing this.

See #59 for more information and a draft pull request.

For my tests I've setup a 1 master 2 worker node cluster. I've encountered weird behaviour after adding node tolerations and not adding this into the CR the storageos container can't connect to it's embedded etcd. Is this a limitation from storageos that single nodes do not work, or did I messed something up?
If I add the toleration back into the CR and redeploy the storageos containers start just fine.

[Arau]

Hi @tobiaskohlbau,

the etcd is a datastore that ensures data consistency across its instances. To do so, it requires quorum. Hence an odd number of nodes. In your case, 2 workers are not enough to have quorum. You need either one or three nodes.
Have you deployed StorageOS in the master? If so, the Pod in the master usually can't start making the discovery process unfit to start the rest of the nodes.

https://docs.storageos.com/docs/platforms/kubernetes/troubleshoot/install#peer-discovery---pod-allocation