Adjustable Commit Timeout

[goraft]

Adjustable Commit Timeout

[xiang90]

@benbjohnson Actually, raft use a false negative approach about the command commits. But I agree we should make this a application decision. I can make the change.

[benbjohnson]

@xiangli-cmu Thanks. I was adding the issue so I wouldn't forget about it. :)

[jvshahid]

Well I don't think waiting indefinitely is an option. We run into this case when a leader lose leadership while there are uncommited log entries in its logs. The next truncate operation will get rid of the log entry and it will never commit causing the command to timeout, if there were no timeout then the command will hang indefinitely, unless we added some sort of timeout+retry logic.

[jvshahid]

I actually wanted to comment on this issue, because i don't know what's the right way to handle this case and it's not apparent from the paper what's the standard behavior. Although the paper say that the state machine should keep track of commands that were processed in case the client reissued a command. So maybe in this case the retry logic should live in the client. What do you guys think ?

[benbjohnson]

@jvshahid Hmmm... the uncommitted logs shouldn't be getting lost just because leadership is lost. They should only get removed if another leader has a different log history. That's a bug that should be fixed.

Removing the timeout should make it hang until either a) the command is committed by that leader, or b) another leader causes the log entry to be removed.

As for retries, I think that should live on the client. If a command fails for some reason, the client is probably the only one with the knowledge to check the current state of the system and optional resubmit the command.

[xiang90]

@benbjohnson I am going to work on this. I will remove the timeout related stuff and let the application make its own decision.

[xiang90]

@benbjohnson BTW, can you review https://github.com/goraft/raft/pull/136/files? The only actual changes is making asnycSend a real one.