ConnectionStateListener inconsistent feedback

[netflix]

ConnectionStateListener inconsistent feedback

[ahfeel]

(I'm using curator 1.0.3)

[Randgalt]

Wow - that seems impossible. The only thing I can think of is that the internal event loop exited. Can you get a thread dump and/or build a test case?

[ahfeel]

Here's a testcase that, from my understanding, should work but doesn't: https://gist.github.com/2157404

It's not exactly the same environment, but the idea is there: 2 servers, kill one to break the Quorum (and the Curator connection here as I only connect to the first server), and I don't get the connection state changes.

[Randgalt]

Dang - I can't reproduce the problem. The test runs fine for me.

[Randgalt]

Describe the environment your test is running in.

[Randgalt]

Hold on - I built a fresh project using the distributed 1.0.3 and I'm reproducing. Stay tuned...

[ahfeel]

Seems strange, I just ran the test again and you're right, it seems to work. But I maintain the original bug, I'm trying to enhance the testcase to make it closer to the original problem, I'll keep you posted.

[ahfeel]

Okay, I'm not 100% sure of the way I'm restarting the ZK server, but the goal is to shutdown a server, break the quorum, and restart it. The RECONNECTED event isn't fired to the listener, and the countdown (3 events) fails.
https://gist.github.com/2163701

[Randgalt]

I'm 90% sure there's a real problem here. But the timing is tricky. I'll keep on it and let you know in the next day or so.

[ahfeel]

Okay great ! Thanks for looking :) Can you just confirm that this new testcase fails for you ?

[Randgalt]

It fails intermittently

[Randgalt]

OK - I've figured it out. The bug is in the test and not Curator. blockUntilConnectedOrTimedOut() returns true if the connection is successful, but false if the connection timeout elapses. I've found that using these in-memory testing clusters can take a long time to connect. The test needs to increase the connectionTimeoutMs and, possibly, just keep retrying until the connection succeeds.

[ahfeel]

I'm pretty sure there's a bug around here. My real use case shows the issue very easily, but is hard to bundle into a single testcase. I'm on IRC (ahfeel) if you have a few minutes to have a look at it :)

[Randgalt]

I'm not an IRC user. But, you can IM me:

Google: randgalt
Yahoo: jordanz