[Feature] Failed backups to not block incoming traffic and trigger high prio alert instead

[gardener]

[Feature] Failed backups to not block incoming traffic and trigger high prio alert instead

[timuthy]

But as long as etcd data is backed by persistent volumes (it is now), we can afford for etcd to continue serve income requests even when backup upload fails

Just wanted to mention one point: In multi-node etcd we have a plan to use Ephemeral volume, if we choose to go with Ephemeral volume then we might lose the data in worst case scenario as data wasn’t backed up.

Wouldn't it make sense to consider the ephemeral volume use-case as an optimized option for the future in order to keep things simple for now? The multi-node project involves more urgent points and requirements like these will only delay a feature roll-out.
WDYT? (/cc @vlerenc)

[dguendisch]

to consider the ephemeral volume use-case as an optimized option for the future in order to keep things simple for now?

fully agree. eph volumes needs I think a certain confidence in overall etcd cluster stability and probably a substantial amount of perf testing and tuning (i.e. what does it mean to now put 20 etcd's data on one single volume and thus share its IOPS).

[gardener-robot]

@amshuman-kr You have mentioned internal references in the public. Please check.

[gardener-robot]

@amshuman-kr You have mentioned internal references in the public. Please check.

[gardener-robot]

@amshuman-kr You have mentioned internal references in the public. Please check.

[vlerenc]

As we have seen lately with the off-by-one (32/33 chunks) on GCP, doesn't it make sense to give this one a higher prio @amshuman-kr?

[ishan16696]

But as long as etcd data is backed by persistent volumes (it is now), we can afford for etcd to continue serve income requests even when backup upload fails

Just wanted to mention one point: In multi-node etcd we have a plan to use Ephemeral volume, if we choose to go with Ephemeral volume then we might lose the data in worst case scenario as data wasn’t backed up.

[ishan16696]

Wouldn't it make sense to consider the ephemeral volume use-case as an optimized option for the future in order to keep things simple for now?

agreed

[vlerenc]

Yes, I would also vote to focus on multi-node/clustered ETCD in the form that can be achieved "the best" (low complexity, low coupling). Ephemeral volumes, as @dguendisch pointed out, require having a solution first and gain trust next, before going there last. Already including that in the challenging task we have at hand, letting it pull in the backup question, raising thereby complexity/coupling even more (leader with failed backups losing leadership sounds like another level of complexity/coupling), sounds like a bit too much too early.

[gardener-robot]

@timuthy You have mentioned internal references in the public. Please check.

[timuthy]

We will follow up with #280 for the readiness probe.
/close