Snakebite doesn't work with HDFS RPC encryption

[elukey]

Snakebite doesn't work with HDFS RPC encryption

[elukey]

I tried very hard to make everything working, but for the moment I am declaring defeated after many unsuccessful tries. The main issue is that the encrypt/decrypt part of python-sasl (and hence cyrus sasl) seems not working for the DIGEST-MD5 encryption. This is an example of what snakebite receives after a successful sasl handshake:

payload: "rspauth=d6141c413171a5eb119aedbbeb8db387"
cipherOption {
  suite: AES_CTR_NOPADDING
  inKey: "\343;p\214\224V\326\026L\367\371d\177\222x\271tf\241N\345\236\206\321F~\000\001\000\000\000\000"
  inIv: "\370l&\317\246\304x.\177\307\336\236C\336\332\225"
  outKey: "\307\214\314\'\355\276A\'\302\360N\261uw\177\347\323T\303\322\342-As\372$\000\001\000\000\000\001"
  outIv: "#\326N\260\237\337s\232N\027N\255L\3265F"
}

The various Iv and Key fields are encrypted, by the Datanode, using the DIGEST-MD5 session keys negotiated via SASL just before (when using auth-conf). I planned to use PyCryptodome for AES encryption but I wasn't able to get to test it.

The first issue was the error described in cyrusimap/cyrus-sasl#614. I found a workaround, namely patching python-sasl, but then I just ended up in a more generic error with more specific hints about what to change/patch.

[elukey]

Updating this issue with some info about testing done. The sasl library was forked to sasl3 (updating bindings etc..) but I wasn't still able to make any RPC to the Namenode to work. The bug that I was getting reported the following error in the Namenode's logs:

INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client 10.64.5.32 threw exception [javax.security.sasl.SaslException: Problems unwrapping SASL buffer [Caused by GSSException: Defective token detected (Mechanism level: Kerberos GSS-API Mechanism Token:Defective Token ID!)]]
javax.security.sasl.SaslException: Problems unwrapping SASL buffer [Caused by GSSException: Defective token detected (Mechanism level: Kerberos GSS-API Mechanism Token:Defective Token ID!)]

After a lot of tests and debugging comparing the code running pure-sasl (working) vs the one usingsasl3 (ending up in the above failure) I noticed that the encoded token sent to the Namenode (to wrap the RPC) was always 4 bytes longer, and the diff was at the front (so the first four bytes were different for some reason, dumping binary was a joy). Google then helped and I found:

https://lists.andrew.cmu.edu/pipermail/cyrus-sasl/2017-March/003002.html

So the extra 4 bytes in front are the length of the payload, added by GSS-API. Cyrus Sasl follows a RFC, Java another one. The trick to make everything to work is to:

• remove the first 4 bytes after encoding with sasl.encode(), and then send to the Namenode.
• add 4 bytes with the len of the token received from the Namenode (something like struct.pack('!I', len(response.token)) + response.token is sufficient).

All the above was to make cyrus sasl to work with RPCs between Client and Namenode, that use GSS-API/kerberos in my case. This makes snakebite to work with calls like ls, but it doesn't when we want to get the content of a file for example, because the code to encrypt the RPC between client and Datanode is still missing. The RPC in fact doesn't use GSS-API, but Digest-MD5, and it follows a different protocol (Datanode protocol).

So next step is to implement the missing code and see if it works. If so, we'll be able to switch snakebite back to cyrus sasl and finally use it in Hadoop clusters with Kerberos and RPC encryption.

[elukey]

As reference https://issues.apache.org/jira/browse/HDFS-6606