typesafehub/constructr-zookeeper

This library enables to use ZooKeeper as cluster coordinator in a ConstructR based cluster

ConstructR-ZooKeeper

This library enables to use ZooKeeper as cluster coordinator in a ConstructR based node.

ConstructR aims at cluster bootstrapping (construction) by using a coordination service and provides etcd as the default one. By means of this library, you will be able to use ZooKeeper as coordination service instead.

Installation

SBT

You will need to add the following dependency in your build.sbt in addition to the core ConstructR ones:

libraryDependencies += "com.lightbend.constructr" %% "constructr-coordination-zookeeper" % "0.4.0"

Gradle

Add following to build.gradle.

Scala 2.11

compile 'com.lightbend.constructr:constructr-coordination-zookeeper_2.11:0.4.0'

Scala 2.12

compile 'com.lightbend.constructr:constructr-coordination-zookeeper_2.12:0.4.0'

Configuration

Check this section in ConstructR for general information about configuration.

Check reference.conf for ZooKeeper related configuration.

Configuring ZK cluster nodes

The default configuration tries to establish a connection to ZooKeeper on localhost:2181.

Override the constructr.coordination.nodes configuration to specify another ZooKeeper node:

constructr.coordination.nodes = ["10.10.10.10:2181"]

The format per node ip:port.

You are also able to connect to a multi-node cluster by specifying multiple nodes, separated by a comma:

constructr.coordination.nodes = ["10.10.10.10:2181", "10.10.10.11:2181", "10.10.10.12:2181"]

Additionally, comma separated connection string format is supported

constructr.coordination.nodes = "10.10.10.10:2181,10.10.10.11:2181,10.10.10.12:2181"

Testing

Run tests:

$ sbt test

constructr-zookeeper's Issues

Upgrade to Scala 2.12.3

MultiJVM tests for constructr-akka-testing failing

I attempted to release this today but seem to be having some 2.11/2.12 bincompat issues. @markusjura have you seen this before?

I get this when running sbt +test:

longshorej@ubuntu1604:~/releases/2017-11-06/constructr-zookeeper$ sbt +test
[info] Loading global plugins from /home/longshorej/.sbt/0.13/plugins
[info] Loading project definition from /home/longshorej/releases/2017-11-06/constructr-zookeeper/project
[info] Set current project to constructr-zookeeper-root (in build file:/home/longshorej/releases/2017-11-06/constructr-zookeeper/)
[info] Setting version to 2.12.4
[info] Reapplying settings...
[info] Set current project to constructr-zookeeper-root (in build file:/home/longshorej/releases/2017-11-06/constructr-zookeeper/)
[info] Updating {file:/home/longshorej/releases/2017-11-06/constructr-zookeeper/}constructr-coordination-zookeeper...
[info] Resolving jline#jline;2.14.5 ...
[info] Done updating.
[info] Updating {file:/home/longshorej/releases/2017-11-06/constructr-zookeeper/}constructr-akka-testing...
[info] Resolving jline#jline;2.14.5 ...
[info] Done updating.
[info] Run completed in 54 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
[info] * com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpec
[JVM-3] Exception in thread "ScalaTest-main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
[JVM-3] 	at org.scalactic.Requirements$RequirementsHelper.macroRequireNonNull(Requirements.scala:184)
[JVM-3] 	at org.scalatest.tools.ArgsParser$.parseReporterArgsIntoConfigurations(ArgsParser.scala:586)
[JVM-3] 	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:890)
[JVM-3] 	at org.scalatest.tools.Runner$.main(Runner.scala:827)
[JVM-3] 	at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-4] Exception in thread "ScalaTest-main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
[JVM-4] 	at org.scalactic.Requirements$RequirementsHelper.macroRequireNonNull(Requirements.scala:184)
[JVM-4] 	at org.scalatest.tools.ArgsParser$.parseReporterArgsIntoConfigurations(ArgsParser.scala:586)
[JVM-4] 	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:890)
[JVM-4] 	at org.scalatest.tools.Runner$.main(Runner.scala:827)
[JVM-4] 	at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-2] Exception in thread "ScalaTest-main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
[JVM-2] 	at org.scalactic.Requirements$RequirementsHelper.macroRequireNonNull(Requirements.scala:184)
[JVM-2] 	at org.scalatest.tools.ArgsParser$.parseReporterArgsIntoConfigurations(ArgsParser.scala:586)
[JVM-2] 	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:890)
[JVM-2] 	at org.scalatest.tools.Runner$.main(Runner.scala:827)
[JVM-2] 	at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-5] Exception in thread "ScalaTest-main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
[JVM-5] 	at org.scalactic.Requirements$RequirementsHelper.macroRequireNonNull(Requirements.scala:184)
[JVM-5] 	at org.scalatest.tools.ArgsParser$.parseReporterArgsIntoConfigurations(ArgsParser.scala:586)
[JVM-5] 	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:890)
[JVM-5] 	at org.scalatest.tools.Runner$.main(Runner.scala:827)
[JVM-5] 	at org.scalatest.tools.Runner.main(Runner.scala)
[JVM-1] Exception in thread "ScalaTest-main" java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;
[JVM-1] 	at org.scalactic.Requirements$RequirementsHelper.macroRequireNonNull(Requirements.scala:184)
[JVM-1] 	at org.scalatest.tools.ArgsParser$.parseReporterArgsIntoConfigurations(ArgsParser.scala:586)
[JVM-1] 	at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:890)
[JVM-1] 	at org.scalatest.tools.Runner$.main(Runner.scala:827)
[JVM-1] 	at org.scalatest.tools.Runner.main(Runner.scala)
[error] Failed: com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpecMultiJvmNode1
[error] Failed: com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpecMultiJvmNode2
[error] Failed: com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpecMultiJvmNode3
[error] Failed: com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpecMultiJvmNode4
[error] Failed: com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpecMultiJvmNode5
[info] com.lightbend.constructr.coordination.zookeeper.MultiNodeZookeeperConstructrSpec
[info] No tests to run for MultiJvm
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[INFO] [11/06/2017 12:44:34.481] [pool-7-thread-2-ScalaTest-running-ZookeeperCoordinationSpec] [akka.actor.ActorSystemImpl(default)] Zookeeper client tries to establish a connection to nodes [127.0.0.1:2181]..
[info] ZookeeperCoordinationSpec:
[info] ZookeeperCoordination
[info] - should correctly interact with zookeeper
[info] ZookeeperNodesSpec:
[info] ZookeeperNodes
[info] - should should extract nodes from a list
[info] - should should extract ZK nodes from a string
[info] Run completed in 3 seconds, 854 milliseconds.
[info] Total number of tests run: 3
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 3, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[error] (constructr-akka-testing/multi-jvm:test) sbt.TestsFailedException: Tests unsuccessful
[error] Total time: 8 s, completed Nov 6, 2017 12:44:36 PM
[INFO] [11/06/2017 12:44:36.149] [default-shutdown-hook-1] [CoordinatedShutdown(akka://default)] Starting coordinated shutdown from JVM shutdown hook
[INFO] [11/06/2017 12:44:36.152] [default-shutdown-hook-1] [CoordinatedShutdown(akka://default)] Starting coordinated shutdown from JVM shutdown hook

Upgrade to constructr 0.17.0

Currently, the constructr version 1.16.1 is used. We should upgrade to the latest version 0.17.0.

Zookeeper nodes should be ephemeral

The znodes created by Akka cluster members are not ephemeral which sometimes leads to the following odd situation :

• Start a new application with a single Akka cluster node using default constructr-zookeeper configuration (max 2 seed nodes)
• The Akka cluster node adds itself to the zookeeper nodes list and everything works as expected
• The list in zookeeper contains the node's actor address (with its current IP address)

Now when the Akka cluster node is restarted, and gets a new IP address, it goes through the usual states and finds its old node address in the nodes list (with the old IP address)

• It tries to use it to join the cluster, but since the IP address is unreachable the node times out and kills the ActorSystem bringing with it the whole application.

If the node created by the Akka cluster node was ephemeral, it would have been cleaned out by zookeeper when the session expired (node went down). So upon restart it would have found no existing nodes and would have registered itself successfully.

Is there any reason why these nodes are not ephemeral ?

Support customization of the root path in Zookeeper

Currently the plugin only supports the path of: /constructr/{clustername}

It would be nice if this was customizable to some extent so that you could do things like:

/someprefix/{clustername}
or
/someprefix/constructr/{clustername}

Clean self-node on disconnect.

I don't believe Self-Nodes should live after disconnect from zookeeper. The library recreates nodes while the curator inter process semaphone mutex uses an ephemeral mechanism.
I believe that the library should be modified so that the nodes are written to zookeeper as ephemeral nodes to tie them to the connection, or at least clean the nodes on shutdown? TTL wasn't a mechanism in zookeeper until recently and isn't needed for service discovery. Basically, if you rip down a cluster, you shouldn't have to wait for the expiry before spinning it back up.

I have a quick example of clean up here (time constraints atm) but I believe ephemeral self-publishing should be considered instead.
https://github.com/jasongoodwin/constructr-zookeeper

Let me know if I'm misunderstanding anything.