Add optional CBOR support (low priority – can disable in Java SDK)

[mhart]

Add optional CBOR support (low priority – can disable in Java SDK)

[bobtiernay-okta]

Would be nice to incorporate https://www.npmjs.com/package/cbor if possible.

[twosider]

For disabling CBOR, see:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/constant-values.html#com.amazonaws.SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY

[mhart]

@schickling sorry, finally got around to doing this! https://github.com/mhart/kinesalite/blob/master/README.md#cbor-protocol-issues-with-the-java-sdk

[alin-simionoiuDE]

now only if we can get tmccombs pull request approved and merged, that would be awesome

[mhart]

Merged and shipped in v3.3.0 – thanks all!

[mhart]

@twosider oh great! thanks. Looks like you can also set an environment variable, AWS_CBOR_DISABLE:

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/SDKGlobalConfiguration.html#AWS_CBOR_DISABLE_ENV_VAR

@IainMeeke let me know if that works for you

[mhart]

Any idea what URL the SDK is trying to call that it's getting a 404 on?

[IainMeeke]

it looks like it is trying to access http://<kinesalite-ip>:4567/

[mhart]

But do you know which action it's throwing that 404 for? ie, which one of these: http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Operations.html

[kellydavid]

I've also encountered similar issues. I've tried it with the ListStreams and PutRecord actions and have had the result that @IainMeeke described above.

[mhart]

I guess this must be something specific to the Java SDK? The latest JavaScript and Python SDKs work fine (and so does the CLI):

$ aws kinesis list-streams --endpoint-url 'http://localhost:4567'
{
    "StreamNames": []
}

I don't know enough about the Java SDK to know what they've changed – does it have debugging capabilities or something?

[kellydavid]

The aws cli works with the actions I mentioned above. It's the sdk that appears to have changed something.

[mhart]

Is there a chance the Java SDK is trying to talk CBOR to kinesalite instead of JSON? I noticed this in the release notes for 1.11.3:

Amazon Kinesis now supports Concise Binary Object Representation (Cbor) data format

Could it be using that by default? (the Content-Type would probably be application/x-amz-cbor-1.1)

This is the first I've heard of Amazon supporting that – kinesalite still only supports application/x-amz-json-1.1 – if this is the case, is there a config setting to change the Java SDK to use JSON?

[kellydavid]

It looks like the 1.11.x sdk does in fact send requests in the application/x-amz-cbor-1.1 format by default

[mhart]

Right, that would be it then. Can you see a way to override that for the SDK at least? It will take a bit of work to get kinesalite to support CBOR

[schickling]

@mhart works for me! Would be great if you could mention it in the Readme :)

[whummer]

+1, AWS_CBOR_DISABLE works for us, too. Thanks for updating.

[Liorba]

Hey can you please share a code snippet on how you used this property?
I'm using kinesis client and it seems that cbor is enabled by
default

[mhart]

@Liorba just set AWS_CBOR_DISABLE in your environment variables. Eg, if you're on bash shell in linux:

export AWS_CBOR_DISABLE=1

Then you should be able to use your kinesis client from that session.

[lyi2012]

It appears to have some validation errors with tmccombs fork, and the a fix is available from https://github.com/lyi2012/kinesalite

[bobtiernay-okta]

https://github.com/tmccombs Thanks for taking this on! @mhart any thoughts on merging the PR?

[teracross]

Note - in order for us to get this working with the Kinesis Client Library we needed to use aws.cborEnabled=false instead in our environment variables.

[mhart]

@teracross can you elaborate a little? with the latest release you shouldn't need to set anything

[teracross]

Sure, this is the maven dependency that we are using for the Kinesis Client Library in our Java 1.8 project:

<dependency>
<groupId>software.amazon.kinesis</groupId>
<artifactId>amazon-kinesis-client</artifactId>
<version>2.1.3</version>
</dependency>

While trying to communicate with an instance of localstack running on Docker on my Mac, we noticed that we were getting 502 errors from LocalStack when trying to communicate with Kinesis. Using Wireshark we determined that the Kinesis Client Library was sending a request to LocalStack using the cbor based protocol. In response, a 502 error was returned to the Kinesis Client Library from localstack.

This might have to do with the version of the Kinesis Client Library that we are using, but just wanted to document this in case anyone else might encounter the same issue.

[mhart]

@teracross are you sure localstack is using the version that has this feature in? It was only released in v3.3.0 2 days ago.

See: #31 (comment)

[teracross]

We are not specifying a LocalStack version, so my assumption is that whichever image that is tagged as "latest" is probably being used.

Was not aware that an update for this specific issue was released 2 days ago, so that is a possibility.

[alin-simionoiuDE]

LocalStack? you are in the wrong place my friend. pretty sure they haven't integrated this change yet