Allow KeepAliveRequest to manage multiple sessions

[kuujo]

Allow KeepAliveRequest to manage multiple sessions

[kuujo]

To be clear, the reason one primitive can slow down other primitives when sharing a client is because a read-heavy primitive that uses SEQUENTIAL read consistency can block command/query responses from a leader in sequencing until a sequential response is received, which often has to wait for entries from the leader since that client has seen recent indexes in other primitives.

[kuujo]

I have been looking in to the idea a bit more tonight, and I think it could be hugely beneficial for both stability and performance. Essentially, this idea evolved out of investigating the thread model in ONOS. What I realized was that the same arguments I made about the threading model in ONOS should apply inside the Copycat client as well. Namely, that we can only guarantee order within a single thread, so the Copycat client is imposing unnecessary overhead on one thread when it has to wait for sequencing for operations that occur another thread.

So, what I'm interested in doing is creating a thread-aware client where it treats each thread as a separate logical session. A logical session adds no overhead - all sessions within a client are still managed with a single register request and keep-alives are shared - but request/response/event sequencing occurs independently of all other sessions, and each logical session appears as a separate session to state machines as well. Each primitive would then use the new thread model to use a specific logical session.

All this would really do is extend ONOS's primitive thread model into Copycat all the way down to the transport layer. That means it would take much better advantage of Netty's concurrency. But more importantly for fault tolerance, it would isolate failures in one thread/primitive from failures in other threads/primitives. For example, if a map put command in one thread gets lost and has to be resent in order to fix sequencing in the cluster, that lost command won't affect/block e.g. a leader election command in another thread until resequencing is complete. It also means sequencing of responses and events will happen independently, so a primitive that's waiting for an event from a follower doesn't block a response from a leader. From what I've seen in the logs, I think this will have a huge impact when the cluster is under high load.