Fix various race conditions

[curator-go]

Fix various race conditions

[basvanbeek]

@meirf: i have not looked into the various race conditions myself as I am not a user of curator anymore.

However I am willing to look into PR's by people which are complemented by unit tests to keep this project alive.

[meirf]

@basvanbeek , any idea on how to resolve this?

I'm seeing this:

panic: close of closed channel
 
goroutine 2849 [running]:
github.com/samuel/go-zookeeper/zk.(*Conn).Close(0xc420795180)
  github.com/samuel/go-zookeeper/zk/conn.go:253 47
github.com/curator-go/curator.(*handleHolder).internalClose(0xc4203058f0, 0xc420302470, 0x0)
  github.com/curator-go/curator/state.go:136 +0x8d
github.com/curator-go/curator.(*handleHolder).closeAndReset(0xc4203058f0, 0xc42587cd00, 0x1e)
  github.com/curator-go/curator/state.go:122 +0x2f
github.com/curator-go/curator.(*connectionState).reset(0xc420302420, 0x1b71d87, 0xf)
  github.com/curator-go/curator/state.go:234 +0x55
github.com/curator-go/curator.(*connectionState).handleExpiredSession(0xc420302420)
  github.com/curator-go/curator/state.go:351 +0xd9
github.com/curator-go/curator.(*connectionState).checkState(0xc420302420, 0xffffff90, 0x0, 0x0, 0xc425ed2600, 0xed0e5250a)
  github.com/curator-go/curator/state.go:318 +0x9c
github.com/curator-go/curator.(*connectionState).process(0xc420302420, 0xc425ed2680)
  github.com/curator-go/curator/state.go:299 +0x16d
created by github.com/curator-go/curator.(*Watchers).Fire
  github.com/curator-go/curator/watcher.go:64 +0x96

[meirf]

I've figured out what the race condition is, but I'm not sure the best solution to submit for PR. Perhaps you can give me a hint if you would be so kind since I'm not as familiar with this library.

Here are the details. Please let me know if I've said anything invalid.

1. The client loses network connectivity to zk.
2. One minute passes.
3. The client regains network connectivity to zk.
4. goroutine A calls s.ReregisterAll() -> Conn() -> checkTimeout() -> reset (bc 1 minute has elapsed) -> closeAndReset() -> conn.Close() which can block for a second
5. goroutine B handles zk.StateExpired (zk cluster sends this bc it considers this client as dead since it didn't ping during 2.) -> reset -> closeAndReset() -> conn.Close() which causes a panic because conn.Close() already closed the connection's c.shouldQuit channel AND s.zooKeeper.getZookeeperConnection was never called by goroutine A because it was blocking for the second so there's no new connection?

I'd be happy to submit a PR if you have me a hint as to appropriate solution.

[meirf]

hey @tevino , I'm not having any luck. Do you have any ideas? Here is a rundown.

[tevino]

I've switched all my projects interacting with Zookeeper from go-curator to a library I made myself, and it turns out much easier than fixing various glitches of a project which nobody maintains actively.

It's a shame that this promising project is rotten.

FYI @meirf