Initialization key should have a TTL

[zalando]

Initialization key should have a TTL

[feikesteenbergen]

The initialize flag means: The cluster has been successfully initialized. (Perhaps renaming the key could be useful). The design being that a cluster which has been succesfully initialized should not be automatically reinitialized.

What we are thinking of is the following:

1. Win race to initialize
2. Set initialize key with a small (minute?) ttl
3. Initialize pgdata
4. Verify correct initialization
5. Update initialize key, remote the ttl

This will ensure that only 1 member will be initializing at any point in time, but if it does not succeed, other members can still initialize.

[jberkus]

Well, the problem with "node has been successfully initialized" is that the key is set at the beginning of initialization. So what about just updating the contents of the key?

Currently the key has: "key":"/service/pgcluster1/initialize","value":"postgresql0"

What about making the value:

{ node: postgresql0, initialization: started, ts: 2015-08-31 19:53:22 }

... and when the initialization is complete, update it to:

{ node: postgresql0, initialization: complete, ts: 2015-08-31 19:54:02 }

The complete initialization should block, but the started initialization should be ignored if it's called again by a node of the same name. Makes sense?

[jberkus]

Hmmmm ... the problem here seems to be more basic.

2015-10-22 18:22:46,989 INFO: removing initialize key after failed attempt to initialize the cluster
Yet:
(patronienv)josh@g:~/patroni$ curl -L http://172.17.0.5:2379/v2/keys/service/batman/initialize
{"action":"get","node":{"key":"/service/batman/initialize","value":"","modifiedIndex":13068,"createdIndex":13068}}

So patroni is actually calling to remove the key, but it's not getting removed. Will dig into why.

[CyberDem0n]

It looks like a bug, which was introduced with #69

[alexeyklyukin]

This problem should be fixed in #77

[jberkus]

This is confirmed as fixed. Closing issue.