☂️ [Feature] Add EtcdMember resource

[gardener]

☂️ [Feature] Add EtcdMember resource

[shreyas-s-rao]

Thanks @timuthy for summarizing the issue.

[timuthy]

/status in-progress

[timuthy]

@amshuman-kr, @shreyas-s-rao and I iterated again over the approach and agreed that we find the EtcdMember design confusing because the resource itself only exists of a status and is not meant to be reconciled by any controller.
Instead, the backup-restore side car can contribute to the corresponding pod.status.conditions with a custom condition type. Extending pod conditions is a supported use-case and enables us to maintain extra status information about each etcd member. Druid can then consult these custom conditions to aggregate the status its AllMemberReady and Ready conditions on etcd.status.conditions.

[amshuman-kr]

#207 describes an alternative of using leases instead of a new EtcdMember resource.

[timuthy]

After working in the problem space we realized that the Lease approach is not ideal as an etcd-member has different status information that should be exposed to etcd-druid in order to take appropriate action, e.g.:

• Is peer-tls enabled for the member -> lets Druid trigger a appropriate rollout if changed from disabled -> enabled (cc @unmarshall)
• Is the disk corrupted -> lets Druid know to take adequate action, e.g. by triggering quorum restoration (cc @abdasgupta)

After a discussion with @unmarshall, we think it's best to revive the idea of the EtcdMember resource instead of continuing maintaining this information in Leases via annotations and other hacks. The design of the EtcdMember API can of course be revisited and revised, e.g. not having a status but another field.

However, there can also be arguments to have .spec and .status fields. For instance, via a .spec it'd be possible for Druid to trigger a data archiving/movement before quorum restoration (cc @vlerenc).

[unmarshall]

For etcd druid to do deterministic recovery/reconciliation its important that individual etcd members publish their up-to-date information. Outside gardener if etcd-druid is not used then any other external actor can be the consumer of this information providing it holistic state of all members in the etcd cluster.

[unmarshall]

/assign

[ishan16696]

Is the disk corrupted -> lets Druid know to take adequate action, e.g. by triggering quorum restoration (cc @abdasgupta)

Just mentioning few points:
When disk get corrupted for 1 cluster member in multi-node etcd cluster, then druid is not getting involved. Please refer to this doc.
When there is a disk failure for majority of cluster member in multi-node etcd cluster, then we have to do the manual intervention as this is the case of permanent quorum loss which is unlikely to occur.

[shreyas-s-rao]

The state of each etcd member is beneficial not only for deciding the order of rolling the members during an update, but also to decide the order of rolling volumes in the case of rolling volumes (due to changes in volume size, storage class, etc). Exact details of how the statefulset can be prepared and then volumes are rolled without downtime for a multi-node etcd cluster are captured in this comment from @unmarshall -> gardener/gardener-extension-provider-aws#646