cloudify/scalazon

Idiomatic, opinionated Scala library for AWS

scalazon's Issues

Enable use of async methods

Amazon SDK exposes XAsync methods (getRecordsAsync etc')

is there a plan on implanting these methods?
it seems like a better approach than simply wrapping normal blocking java code with scala async.

I would love to see mistakes in my arguments.

Kindly,

Make this available in Maven Central

We try to avoid custom repos as much as possible (company policy)

Planning ticket for adding in async DynamoDB library

Hey @cloudify - as you know we are big fans of Scalazon at Snowplow.

We are starting to do more with DynamoDB, initially related to its usage by the KCL, and want to use the Scala async-dynamodb project. Problem is, it's abandonware, with a live but unhosted fork here; we also did a fork a couple of years back.

I wondered if you had any appetite to support DynamoDB in scalazon? We could pull in the live fork, preserving its git history. I'm happy to do the heavy lifting to port it over - it's really important to Snowplow to have a robust idiomatic Scala lib for DynamoDB...

Let me know your thoughts...

Not possible to specify startingSequenceNumber for getIterator call

Although you are allowed to create iterators with type "AFTER_SEQUENCE_NUMBER", it looks like you are unable to specify a starting sequence number, this makes it difficult to implement checkpointing.

getShardIteratorRequest never makes a call to setStartingSequenceNumber

Incomplete data retrieved.

Hi Federico,

I've modified your example to add 10 items to Kinesis and then attempt to retrieve them 10 separate times to illustrate a bug I'm seeing in some of my other projects where not all of the records are retrieved:

• snowplow/kinesis-example-scala-consumer#1
• snowplow/kinesis-example-scala-producer#4

Do you know if this is expected behavior from Kinesis?
Am I doing something subtly wrong when retrieving the data?

Below, some attempts retrieve all 10 records, but other attempts, such as the last one, read less than 10 records.

stream created
stream active
ACTIVE
true
data stored: shardId-000000000000, 49535162123327736836226361226397834013393393790669029377
data stored: shardId-000000000000, 49535162123327736836226361326022690278750787438945239041
data stored: shardId-000000000000, 49535162123327736836226361368991565008786413418608328705
data stored: shardId-000000000000, 49535162123327736836226362473531453190127716825632342017
data stored: shardId-000000000000, 49535162123327736836226362521162196476315335329660796929
data stored: shardId-000000000000, 49535162123327736836226362486934026452529592727012114433
data stored: shardId-000000000000, 49535162123327736836226362545355462718347236243547357185
data stored: shardId-000000000000, 49535162123327736836226362496631199651551572522497474561
data stored: shardId-000000000000, 49535162123327736836226362425675822156409032816724017153
data stored: shardId-000000000000, 49535162123327736836226363990199997461027351869823909889
Retrieving data, attempt 1
data retrieved
==Record chunk.
Read 0 records.

Retrieving data, attempt 2
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
Read 3 records.

Retrieving data, attempt 3
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362486934026452529592727012114433
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362496631199651551572522497474561
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362521162196476315335329660796929
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362545355462718347236243547357185
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226363990199997461027351869823909889
data: hello
partitionKey: k1
Read 10 records.

Retrieving data, attempt 4
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
Read 5 records.

Retrieving data, attempt 5
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
Read 5 records.

Retrieving data, attempt 6
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
Read 5 records.

Retrieving data, attempt 7
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362486934026452529592727012114433
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362496631199651551572522497474561
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362521162196476315335329660796929
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362545355462718347236243547357185
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226363990199997461027351869823909889
data: hello
partitionKey: k1
Read 10 records.

Retrieving data, attempt 8
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362486934026452529592727012114433
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362496631199651551572522497474561
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362521162196476315335329660796929
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362545355462718347236243547357185
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226363990199997461027351869823909889
data: hello
partitionKey: k1
Read 10 records.

Retrieving data, attempt 9
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362425675822156409032816724017153
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362473531453190127716825632342017
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362486934026452529592727012114433
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362496631199651551572522497474561
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362521162196476315335329660796929
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226362545355462718347236243547357185
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226363990199997461027351869823909889
data: hello
partitionKey: k1
Read 10 records.

Retrieving data, attempt 10
data retrieved
==Record chunk.
sequenceNumber: 49535162123327736836226361226397834013393393790669029377
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361326022690278750787438945239041
data: hello
partitionKey: k1
sequenceNumber: 49535162123327736836226361368991565008786413418608328705
data: hello
partitionKey: k1
Read 3 records.

stream deleted

Update dependencies to pull official AWS SDK

Now that Kinesis is GA we can add the library to the dependencies.

Any way to set the region for Kinesis?

Looked thru the API and I failed to find a parameter I can set or flip a value to change the region that Kinesis is running. It looks like it defaults to us-east-1. Any suggestions to setting the DEFAULT_KINESIS_ENDPOINT? Thanks for the lib, its been very useful.

Java API reference:
http://docs.aws.amazon.com/kinesis/latest/dev/kinesis-using-api-java.html#kinesis-using-api-create-stream

Code Snippet:
client = new AmazonKinesisClient();
client.setEndpoint(endpoint, serviceName, regionId);

Modify example to show how to get a PutResult after putting a data and key.

Sorry, I've been reading through your source, but wasn't able to get this working. I'm probably overlooking something simple because I'm confused where calling put
returns a Future[Unit] rather than a Future[PutResult].

Clarify in README in which repo this library is hosted