webdevops/azure-k8s-autopilot

K8S operator for Azure VMSS/VM for automatic repair and update

azure-k8s-autopilot's Issues

go stack trace when losing election

Hi,
with an image built with "master", I receive this message when using lease with 2 pods.
BTW, lease feature is not enable by default in last docker image. I had to use LEASE_ENABLE=true

this is log of the election loser. The other one won, and works properly.
{"level":"info","caller":"autopilot/main.go:360","msg":"trying to become leader"}
[controller-runtime] log.SetLogger(...) was never called, logs will not be displayed:
goroutine 97 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
sigs.k8s.io/controller-runtime/pkg/log.eventuallyFulfillRoot()
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/log.go:59 +0xbd
sigs.k8s.io/controller-runtime/pkg/log.(*delegatingLogSink).Enabled(0xc0003c64c0, 0xedc939edd?)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/log/deleg.go:111 +0x3d
github.com/go-logr/logr.Logger.Enabled(...)
/go/pkg/mod/github.com/go-logr/[email protected]/logr.go:261
github.com/go-logr/logr.Logger.Info({{0x1b6d368?, 0xc0003c64c0?}, 0x1b71d80?}, {0x190fddc, 0x18}, {0x0, 0x0, 0x0})
/go/pkg/mod/github.com/go-logr/[email protected]/logr.go:274 +0x78
github.com/operator-framework/operator-lib/leader.Become({0x1b6a490, 0xc000138028}, {0xc00004e03b, 0x25}, {0x0, 0x0, 0x0?})
/go/pkg/mod/github.com/operator-framework/[email protected]/leader/leader.go:192 +0xa0e
github.com/webdevopos/azure-k8s-autopilot/autopilot.(*AzureK8sAutopilot).leaderElect(0xc000250840)
/go/src/github.com/webdevops/azure-k8s-autopilot/autopilot/main.go:369 +0x167
github.com/webdevopos/azure-k8s-autopilot/autopilot.(*AzureK8sAutopilot).Start.func1()
/go/src/github.com/webdevops/azure-k8s-autopilot/autopilot/main.go:248 +0x26
created by github.com/webdevopos/azure-k8s-autopilot/autopilot.(*AzureK8sAutopilot).Start
/go/src/github.com/webdevops/azure-k8s-autopilot/autopilot/main.go:247 +0x56

Lock is stored in the node resource which may disappear

see https://kubernetes.io/docs/concepts/architecture/cloud-controller/#node-controller
If the VM is removed from the cloud, the cloud-controller-manager will remove the node from kubernetes.
azure-k8s-autopilot will lose the lock timing value and continue with the next VM, which is not desirable.

Find another place to store these locks if possible.

Add option to drain with disable eviction if drain fails

When PDBs are wrongly configured, drain can fail and cause pods to block the node upgrade.
Add an option that retries the drain with --disable-eviction if the first 'classic' one failed after a timeout.

Relates to #6
See kubernetes/kubernetes#83307 (comment)

Update shoutrrr to the latest release

shoutrrr added support for Google Chat recently, which I'd like to receive my notifications on.

Update from v0.2 to v0.5.2.

Crashes on API server timeout

When the API server is not reachable (timeout) for any reason, this controller crashes. It's then restarted and will resume its work, so it's not a critical issue. I think some work can be done to retry with backoff instead of crashing. I'll post some logs in there when I reproduce the issue.

Add option to drain with disable eviction

Add a drain configuration option to use --disable-eviction kubectl flag.

Is this project still maintained?

I'd love to use this project for my infrastructure. Seeing that there were no updates since last year, do you still use it? Will you continue maintaining it? I would love to contribute on it if I find anything to do.