can't remove leader by method of RaftHandle's removeServer

[jackjoesh]

can't remove leader by method of RaftHandle's removeServer

[belaban]

RaftHandle.removeServer() is only for changing the (static) membership of a cluster. You cannot transfer the leader role to any given member, as this might violate the Raft properties, ie. the member with the longest log *and* the majority of votes becomes leader. If you for example transferred leadership to a member who doesn't have the longest log, you will lose *committed* log entries, and this violates the Raft prototocol.

…

On 19.07.23 21:32, jackjoesh wrote: I want to manually transfer leader at sometime. Thus, I invoke removeServer method of RaftHandle in leader node to remove itself, but it seems not work. How can I remove current leader or transfer my leader in my code? thanks — Reply to this email directly, view it on GitHub <#212>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADPZXIIKU3PA2EI43PRBSDXRAY5LANCNFSM6AAAAAA2QMJSY4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- Bela Ban | http://www.jgroups.org

[jackjoesh]

RaftHandle.removeServer() is only for changing the (static) membership of a cluster. You cannot transfer the leader role to any given member, as this might violate the Raft properties, ie. the member with the longest log and the majority of votes becomes leader. If you for example transferred leadership to a member who doesn't have the longest log, you will lose committed log entries, and this violates the Raft prototocol.
…
On 19.07.23 21:32, jackjoesh wrote: I want to manually transfer leader at sometime. Thus, I invoke removeServer method of RaftHandle in leader node to remove itself, but it seems not work. How can I remove current leader or transfer my leader in my code? thanks — Reply to this email directly, view it on GitHub <#212>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADPZXIIKU3PA2EI43PRBSDXRAY5LANCNFSM6AAAAAA2QMJSY4. You are receiving this because you are subscribed to this thread.Message ID: @.***>
-- Bela Ban | http://www.jgroups.org

Yes, I know, we can't violate the raft protocol.
But we have some special requirements: we have our own health-check logic in each node.
If current leader node is not health, we want to force current leader node to give up its own leader role and transfer to other nodes. So shall we have two ways:
1 support a transfer leader command ,and regard as a type of raft event log to add to state machine, so I can guarantee consistency
2 remove current leader node from the cluster, and add to the cluster after new leader starting as a follower
If we can't support 1 way, can we support 2 way?
Thanks

[jabolina]

Approach 1 sounds like creating an external election algorithm. So I highly suggest 2.

The current ELECTION protocol uses the JGroups views to trigger the election mechanism. To trigger the election, the current RAFT leader disconnects from the cluster and connects back. This should cause a view change, and an election round starts. You could use the JChannel#disconnect and JChannel#connect(String) methods. The channel is on the RaftHandle#channel method. You wouldn't need to change the RAFT membership with this. But I suspect that depending on the timing, the same leader could be re-elected.

[jackjoesh]

Approach 1 sounds like creating an external election algorithm. So I highly suggest 2.

The current ELECTION protocol uses the JGroups views to trigger the election mechanism. To trigger the election, the current RAFT leader disconnects from the cluster and connects back. This should cause a view change, and an election round starts. You could use the JChannel#disconnect and JChannel#connect(String) methods. The channel is on the RaftHandle#channel method. You wouldn't need to change the RAFT membership with this. But I suspect that depending on the timing, the same leader could be re-elected.

thank you, let me try disconnecting and connecting again. By the way , do we try this way before? I remember maybe it throws "channel close" exception after disconnecting and reconnecting again

[belaban]

If you disconnect, then wait a few milliseconds before reconnecting, another leader should have been elected (f you still have a majority). If you then reconnect, the old leader should not become leader.

[belaban]

Just call JChannel.disconnect(), not JChannel.close(). In the first case, JChannel.connect() will succeed, in the latter it will fail with a ChannelClosedException

[jackjoesh]

Just call JChannel.disconnect(), not JChannel.close(). In the first case, JChannel.connect() will succeed, in the latter it will fail with a ChannelClosedException

it seems still has some problem, After disconnecting, and reconnect again , it occurs the following exception. Is my invocation fault?

        this.raftHandle.channel().disconnect();
        Thread.sleep(1000l);
        this.raftHandle.channel().connect(this.raftClusterName);

org.iq80.leveldb.DBException: Closed
at org.fusesource.leveldbjni.internal.JniDB.iterator(JniDB.java:100) ~[leveldbjni-all-1.8.jar:1.8]
at org.fusesource.leveldbjni.internal.JniDB.iterator(JniDB.java:95) ~[leveldbjni-all-1.8.jar:1.8]
at org.jgroups.protocols.raft.LevelDBLog.sizeInBytes(LevelDBLog.java:217) ~[jgroups-raft-1.0.11.Final.jar:?]
at org.jgroups.raft.util.LogCache.sizeInBytes(LogCache.java:206) ~[jgroups-raft-1.0.11.Final.jar:?]
at org.jgroups.protocols.raft.RAFT.logSizeInBytes(RAFT.java:378) ~[jgroups-raft-1.0.11.Final.jar:?]
at org.jgroups.protocols.raft.RAFT.start(RAFT.java:570) ~[jgroups-raft-1.0.11.Final.jar:?]
at org.jgroups.stack.ProtocolStack.startStack(ProtocolStack.java:890) ~[jgroups-5.2.14.Final.jar:5.2.14.Final]
at org.jgroups.JChannel.startStack(JChannel.java:919) ~[jgroups-5.2.14.Final.jar:5.2.14.Final]
at org.jgroups.JChannel._preConnect(JChannel.java:797) ~[jgroups-5.2.14.Final.jar:5.2.14.Final]
at org.jgroups.JChannel.connect(JChannel.java:322) ~[jgroups-5.2.14.Final.jar:5.2.14.Final]
at org.jgroups.JChannel.connect(JChannel.java:316) ~[jgroups-5.2.14.Final.jar:5.2.14.Final]

[jabolina]

Try something like:

        this.raftHandle.channel().disconnect();
        Thread.sleep(1000l);
        this.raftHandle.raft().log(null); // <--- new line
        this.raftHandle.channel().connect(this.raftClusterName);

This should cause LevelDB to reinitialize after connecting again.

[jackjoesh]

this.raftHandle.raft().log(null); // <--- new line

cool, it works now, thanks