Zookeeper SASL Vs Curator

[netflix]

Zookeeper SASL Vs Curator

[Randgalt]

Might you be able to put together a test (or instructions) on how to reproduce this?

[Randgalt]

Are you calling authorization() in the CuratorFrameworkFactory builder?

[antonio01]

Hi,

I can't use .authorization( because the Zookeeper implementation prevents the setting of the sasl scheme.

See SASLAuthenticationProvider:24 :

public KeeperException.Code
handleAuthentication(ServerCnxn cnxn, byte[] authData)
{
// Should never call this: SASL authentication is negotiated at session initiation.
// TODO: consider substituting current implementation of direct ClientCnxn manipulation with
// a call to this method (SASLAuthenticationProvider:handleAuthentication()) at session initiation.
return KeeperException.Code.AUTHFAILED;

}

Also I've used the Zookeeper Client too and it works.

I'm writing some notes to post how to reproduce it.

[antonio01]

Hi,

Following the guide on https://cwiki.apache.org/ZOOKEEPER/zookeeper-and-sasl.html

I've setup the Zookeeper Server to use SASL along with java security:

into the server conf/zoo.cfg:

authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider

jaasLoginRenew=3600000

Then place somewhere a file jaas.conf with the following content (I'm using DIGEST-MD5 authentication):

Server {
org.apache.zookeeper.server.auth.DigestLoginModule required
user_super="adminsecret"
user_bob="bobsecret";
};

and into conf/java.env set the path to the jaas.conf

SERVER_JVMFLAGS="-Djava.security.auth.login.config=/path/to/server/jaas/file.conf"

(those settings are for the sh scripts. For windows the SERVER_JVMFLAGS should be set into the zkEnv.cmd and then the variable passed into the zkServer.cmd just after the CLASSPATH var at the java call.

After that you can start up the ZK server.

For the client side, create a new file jaas_client.conf with

Client {
org.apache.zookeeper.server.auth.DigestLoginModule required
username="bob"
password="bobsecret";
};

Now Use a Zookeeper client to do any operation against the server. Before creating the client do:

System.setProperty("java.security.auth.login.config","/path/to/server/jaas/file.conf");

and then create the zk client.

In the server log you will see:

[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2183:SaslServerCallbackHandler@130] - Setting authorizedID: bob
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2183:ZooKeeperServer@934] - adding SASL authorization for authorizationID: bob

This happens with Curator as well but after that the ConnectionLoss on the client side.

[antonio01]

Have you ever tried to use Curator with an authentication mechanism that involves using a jaas.conf file?
Which authentication mechanism have you tried?

[Randgalt]

I tried the example as described above and everything seems to work correctly. Here's the code I used:

        Timing                  timing = new Timing();
        System.setProperty("java.security.auth.login.config","/path/to/jaas.conf");
        CuratorFramework        client = CuratorFrameworkFactory.newClient("localhost:2181", timing.session(), timing.connection(), new RetryOneTime(1));

        client.start();
        client.checkExists().forPath("/hey");

        log.debug("about to sleep");
        Thread.sleep(100000);

The client sleeps and the server says "Successfully authenticated client: authenticationID=bob; authorizationID=bob".

[Randgalt]

Maybe you aren't setting a long enough connection timeout?

[antonio01]

Hi,

Can you add a watcher while the client adds a node? I don't receive any event.
Settting the corretc ACL I'm able to create the node but I cannot receive from another client the events.

Also I cannot make work a ServiceDiscovery (which works without sasl):

System.setProperty("java.security.auth.login.config","/path/to/jaas.conf");
ServiceDiscovery discovery = ServiceDiscoveryBuilder
.builder(String.class).basePath(root).client(client).build();

    try {
        discovery.start();

        Collection<String> names = discovery.queryForNames();

It hangs on discovery.queryForNames();

The same works if I use the digest schema since in that case I can add to the client
.authorization("digest", "username:pwd".getBytes()

Thank you.

[Randgalt]

I just tried it with ServiceDiscovery and queryForNames() and it works fine. I have a strong feeling that your connection and/or session timeouts are too low.

[antonio01]

Which settings did you use?

I've even set
CuratorFramework client = CuratorFrameworkFactory.newClient("localhost:2181", 100000, 100000, new RetryOneTime(1));

but still no luck.

Are you talking about another session/connection timeout?

[antonio01]

Ok, some updates. It seems related to the number of operations I execute with the same client.
Indeed even the discovery works if it's the only one thing a do with a client just created.

Can you try the below code? I've extended your example to do more operations.

Example:

Timing timing = new Timing();
System.setProperty("java.security.auth.login.config","/path/to/jaas.conf");
CuratorFramework        client = CuratorFrameworkFactory.newClient("localhost:2182", timing.session(), timing.connection(), new RetryOneTime(1));

client.start();
client.checkExists().forPath("/hey");

client.create().withMode(CreateMode.EPHEMERAL).withACL(Ids.CREATOR_ALL_ACL).forPath("/hey", new byte[0]);
client.create().withMode(CreateMode.EPHEMERAL).withACL(Ids.CREATOR_ALL_ACL).forPath("/again", new byte[0]);


Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
    at com.netflix.curator.ConnectionState.getZooKeeper(ConnectionState.java:84)
    at com.netflix.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:90)
    at com.netflix.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:381)
    at com.netflix.curator.framework.imps.CreateBuilderImpl$6.call(CreateBuilderImpl.java:356)
    at com.netflix.curator.framework.imps.CreateBuilderImpl$6.call(CreateBuilderImpl.java:336)
    at com.netflix.curator.RetryLoop.callWithRetry(RetryLoop.java:85)
    at com.netflix.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:332)
    at com.netflix.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:294)
    at com.netflix.curator.framework.imps.CreateBuilderImpl$2.forPath(CreateBuilderImpl.java:149)
    at com.netflix.curator.framework.imps.CreateBuilderImpl$2.forPath(CreateBuilderImpl.java:115)

[Randgalt]

Persistence pays off! I've found the problem. The curator event processor wasn't handling the SaslAuthenticated event. It was treating it as a disconnection. I really appreciate your help on this. I'll try to build new JARs today or tomorrow.

[antonio01]

Many Thanks!

[mbbui]

Which version this fix is in? Thanks.

[Randgalt]

1.1.8/1.0.9 - April 17, 2012