Ex-master cannot return come back after net-split

[zalando]

Ex-master cannot return come back after net-split

[alexeyklyukin]

It looks like the old master has advanced its WAL position past the promotion point of a new master. I think it might happen during the network outage, when a new data has been written into the master's WAL, but has not been propagated to replicas (this is also possible in synchronous mode). You can configure Patroni to call pg_rewind in order to bring the former master up-to-date.

[katoquro]

Yep, you are right. But this broken node can became a master if there are no competitors

2015-11-18 12:05:50,027 INFO: Lock owner: node5; I am node5
2015-11-18 12:05:50,028 INFO: no action.  i am the leader with the lock
172.17.0.101 - - [18/Nov/2015 12:05:52] "OPTIONS / HTTP/1.0" 200 -

[alexeyklyukin]

Well, if all other nodes in the cluster have died, then promoting a single leftover node to a master is a sane thing to do, isn't it?

[katoquro]

I'm not sure because this node can be outdated and can contain inconsistent data
Such cases can corrupt logic on the clients so it will better to shutdown such nodes.

[alexeyklyukin]

It's not a task of Patroni to detect such 'broken' nodes. Your monitoring system should do it (based, for instance, on the replication lag), and it should be human decision to shut them down.

[drnic]

I'd like it to the be the task of Patroni to automate and make these decisions

On Wed, Nov 18, 2015 at 6:05 AM, Oleksii Kliukin [email protected]
wrote:

It's not a task of Patroni to detect such 'broken' nodes. Your monitoring system should do it (based, for instance, on the replication lag), and it should be human decision to shut them down.

Reply to this email directly or view it on GitHub:
#99 (comment)

[drnic]

Is it entirely out of scope for Patroni cells to self administer this? The only orchestration can be external?

On Wed, Nov 18, 2015 at 5:20 AM, katoquro [email protected]
wrote:

I'm not sure because this node can be outdated and can contain inconsistent data

Such cases can corrupt logic on the clients so it will better to shutdown such nodes.

Reply to this email directly or view it on GitHub:
#99 (comment)

[alexeyklyukin]

You can use pg_rewind and avoid this problem altogether.

[alexeyklyukin]

There are other cases of a node that is unable to join the cluster (for instance, if replication username/password is incorrect). It's not possible/does not make much sense to detect every issue like this by Patroni - it should be a task of the monitoring system to realize that some replicas are potentially unhealthy and then a human interaction to fix it.

[drnic]

@alexeyklyukin thanks for pointing me to pg_rewind