Serpent implementation output not match to MCrypt implementation

[bcgit]

Serpent implementation output not match to MCrypt implementation

[timw]

In general if you can provided a test vector (plaintext, iv, key, ciphertext) that fails it's a lot easier to help with these kinds of issues.

It looks like the Serpent implementation in mcrypt (or the PHP binding of mcrypt) is broken.

Using serpent/ecb test vectors from the Serpent AES submission (cases I=1, I=2, I=3 from floppy4/ecb_vt.txt) the mcrypt Serpent fails all of them. BC Java (and I presume C# as it shares test vectors) passes all of them.
The mcrypt Twofish and Rijndael implementations pass the same vectors as BC Java, so this seems isolated to the Serpent implementation.

The PHP snippet below demonstrates the failure on the serpent test cases (these are serpent/ecb, so no CFB mode or padding involved).

Tests were run using mcrypt: stable 2.5.8, php54-mcrypt: stable 5.4.26, and php54: stable 5.4.26 from Homebrew.

<?php
function crypt_test($algo, $mode, $key, $iv, $name, $plaintext, $expected) {
    $cipher = mcrypt_module_open($algo,'',$mode,'');
    mcrypt_generic_init($cipher, $key, $iv);
    $encrypted = mcrypt_generic($cipher, hex2bin($plaintext));

    printf("%s/%s %s:\n expected: %s\n   actual: %s\n", $algo, $mode, $name, bin2hex($encrypted), $expected);

    mcrypt_generic_deinit($cipher);
    mcrypt_module_close($cipher);
}

$key = hex2bin('00000000000000000000000000000000');
$iv = hex2bin('00000000000000000000000000000000');

crypt_test('serpent', 'ecb', $key, $iv, 'zeros', '00000000000000000000000000000000', 'e9ba668276b81896d093a9e67ab12036');
crypt_test('serpent', 'ecb', $key, $iv, 'I1', '80000000000000000000000000000000', '10b5ffb720b8cb9002a1142b0ba2e94a');
crypt_test('serpent', 'ecb', $key, $iv, 'I2', '40000000000000000000000000000000', '91a7847ef1cd87551b5b4bf6f8e96e2c');
crypt_test('serpent', 'ecb', $key, $iv, 'I3', '20000000000000000000000000000000', '5d32aece8383fb2ee22cb4a6061d1429');

?>

[pkorsukov]

Thank you. It seems to be a PHP problem. I have submitted bug for them https://bugs.php.net/bug.php?id=66949. You can close the ticket. Thanks!

[drizzt]

I have the same problem and it's a BouncyCastle bug!
gnu-crypto, libgcrypt, mcrypt, etc, returns another value

[dghgit]

Then you need to advise gnu-crypto, libgcrypt, mcrypt, etc, returns another value to fix their implementations. Please see http://www.bouncycastle.org/jira/browse/BMA-52

Unless this is in relation to something new this is not a bug in Bouncy Castle.

[dghgit]

I have confirmed this is the usual problem - just to repeat what has already been said, the AES submission for Serpent is available at:

http://www.cl.cam.ac.uk/~rja14/Papers/serpent.tar.gz

Bouncy Castle's version of Serpent passes these, we also confirmed with the authors of Serpent in 2009 that the AES submission showed the test vectors in correct ordering.

[drizzt]

You are right. The other cryptographics libreries uses the NESSIE format.
You are the only one I found using the correct (aka original) format.
Since I need compatibity with pre-existant code I'll reverse the input and output arrays of byte.

Thank you for the explanation

Okay, just to follow up on this one, it turns out that there was some sort of misunderstanding when we first approached the Serpent authors for clarification. Apparently we should be using the NESSIE vectors. Funnily this confusion is not rare, and the official name for the other version is Tnepres. We're trying to sort it out now.