scanamo/scanamo

Simpler DynamoDB access for Scala

Scanamo

Scanamo is a library to make using DynamoDB with Scala simpler and less error-prone.

The main focus is on making it easier to avoid mistakes and typos by leveraging Scala's type system and some higher level abstractions.

Installation

libraryDependencies += "org.scanamo" %% "scanamo" % "1.0.0"

Basic Usage

scala> import org.scanamo._
scala> import org.scanamo.syntax._
scala> import org.scanamo.generic.auto._
 
scala> val client = LocalDynamoDB.client()
scala> import software.amazon.awssdk.services.dynamodb.model.ScalarAttributeType._
scala> val farmersTableResult = LocalDynamoDB.createTable(client)("farmer")("name" -> S)

scala> case class Farm(animals: List[String])
scala> case class Farmer(name: String, age: Long, farm: Farm)
scala> val table = Table[Farmer]("farmer")

scala> val ops = for {
     |   _ <- table.putAll(Set(
     |       Farmer("McDonald", 156L, Farm(List("sheep", "cow"))),
     |       Farmer("Boggis", 43L, Farm(List("chicken")))
     |     ))
     |   mcdonald <- table.get("name" -> "McDonald")
     | } yield mcdonald
scala> Scanamo.exec(client)(ops)
res1: Option[Either[error.DynamoReadError, Farmer]] = Some(Right(Farmer(McDonald,156,Farm(List(sheep, cow)))))

Note: the LocalDynamoDB object is provided by the scanamo-testkit package.

For more details, please see the Scanamo site.

License

Scanamo is licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

scanamo's Issues

question re: why scanConsistent and queryConsistent don't exist in the Async interface

ScanamoFree seems to have methods for strongly consistent reads but scan (scanConsistent) and and query(queryConsistent) version of these are not included in ScanamoAsync (but the get is)

What is the reasoning behind this?

No DynamoFormat for Byte or Byte[]

When trying to use the related scanamo-scrooge, I tried to use thrift objects backed by the byte thrift type. Then since DynamoFormat doesn't include support for the byte type, I can't use existing thrirft objects with scanamo-scrooge. Finally thrift also supports a binary type which is backed by a Byte[]. Then based on comments by @philwills I tried to implement the type transformations for Byte and Byte[], but my scala-fu is lacking. A related issue with details is filed: guardian/scanamo-scrooge#4 .

Batch GET requests to scanamo fail with duplicate keys

Currently if a 'getAll' method is called is a non distinct list this is the error we see:
AWS Error Code: ValidationException, AWS Error Message: Provided list of item keys contains duplicates

Scanamo 1.0

Following on from #134, I'd like to get out a release based on Cats 1.0.0-MF. My plan is to use the same version for Scanamo, so it would also be 1.0.0-MF to try and avoid confusion. Scanamo is reasonably fully featured and stable now, so whilst I'd like to move to things like #135, I'm not planning on making that kind of large change in this timeframe.

Apart from moving to the latest versions of dependencies, the only change I'm planning is to deprecate all methods in Scanamo and ScanamoAsync apart from exec. Everything they do can be achieved without much more ceremony (less if more than one operation issued) using Table which is more flexible, so I'd like to make it clear that that's the preferred method.

I'm very keen to hear any feedback on this approach, but want to move forward to a release quickly.

who's Marley

Really interested in this lib, the README says

Marley is licensed under the Apache License, Version 2.0

Who's Marley?

just so you know... async doesn't really mean async

http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/AmazonDynamoDBAsyncClient.html

"Async" you say, Amazon? Oh, well, I guess that means you'll be using the NIO
HTTP clients?

https://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/protocol/HttpRequestExecutor.html

Which, and I quote, "is a client side HTTP protocol handler based on the blocking (classic) I/O model."

Possible options:

• https://github.com/alessandrosimi/aws-dynamodb-nio

How to query a table with hash and range key

How do I query a table with a hash and range key, while only passing the hash key?

disable automatic derivation

My team prefers semi-auto derivation as per https://github.com/fommil/stalactite

It would be great if you followed the pattern for derivation laid out in https://github.com/milessabin/export-hook to

1. avoid lots of corner cases you may not have considered
2. allow users to opt-in to full generic derivation

thanks for a great library!

AWS SDK for DynamoDB should be a provided dependency?

Hey there,

Just wondering if we should make the SDK for DynamoDB a provided dependency. I find myself doing the following in projects:

libraryDependencies ++= {
  Seq(
    "com.gu"                  %% "scanamo"                    % "0.9.5" exclude("com.amazonaws", "aws-java-sdk-dynamodb"),
    "com.amazonaws"            % "aws-java-sdk"               % "1.11.158"
  )
}

The reason for doing this is usually to include other AWS libraries or for the purposes of tracking metrics on the DynamoDB client which requires other components of the SDK

AwsSdkMetrics.enableDefaultMetrics
AwsSdkMetrics.setCredentialProvider(DefaultAWSCredentialsProviderChain.getInstance)
AwsSdkMetrics.setMetricNameSpace(settings.accountsTableName.getOrElse("my-app-name"))

Wondering if you had any thoughts :-)

Rename to Guantanamo

Possibly the best name I have ever come up with for somebody else's project.

Come on, just think of the package name: com.gu.antanamo!

The wrapper should be able to use filter expression in query

At the moment, the query operation has support only for key expression. It would be great to have add a filter expression functionality as well.

Incrementing and Decrementing Numeric Attributes

I want to perform a DynamoDB update operation like this http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithItems.html#WorkingWithItems.AtomicCounters

   // Increment an atomic counter
    const params = {
        TableName:table,
        Key:{
            "idName": idName
        },
        UpdateExpression: "set countVal = countVal + :val",
        ExpressionAttributeValues:{
            ":val":1
        },
        ReturnValues:"UPDATED_NEW"
    };

How do I increment & decrement attributes in Scanamo? http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.UpdateExpressions.html#Expressions.UpdateExpressions.SET.IncrementAndDecrement

Enum value types mapped incorrectly when used directly in queries

When an enumeration is defined as sealed trait + case objects hierarchy, using enum values directly in update operations results in record being corrupted because of incorrect mapping. Sample below fails with PropertyReadError(status,TypeCoercionError(java.lang.Exception: {M: {},} is not a recognised member of the Enumeration)) error.

object EnumTest {
  sealed trait Status
  case object Pending extends Status
  case object Completed extends Status

  case class Record(id: String, status: Status)
}

class EnumTest extends FunSpec with Matchers {
  import EnumTest._

  lazy val client = AmazonDynamoDBClientBuilder.standard().withEndpointConfiguration(new EndpointConfiguration("http://localhost:8000", "")).build()
  lazy val table = Table[Record]("TestTable")

  describe("Scanamo") {
    val id = "ABC"
    val putRequest = table.put(Record(id, Pending))
    Scanamo.exec(client)(putRequest)

    it("should update enum value") {
      val updateRequest = table.update('id -> id, set('status -> Completed))
      val updated = Scanamo.exec(client)(updateRequest)
      updated.right.value should equal(Record(id, Completed))
    }
  }
}

Similarly, enum values can't be used directly in conditional clauses.

When enum is casted to base trait, either by .asInstanceOf[Status] or val status: Status = , everything works as expected.

Support manual pagination keys

Currently there's no way to manually specify pagination params when querying -- would be nice to have some way of getting the lastEvaluatedKey of a query, and being able to pass an exclusiveStartKey.

Implement 'Between' RangeKeyCondition

Similar to BeginsWith it would be great to have a Between RangeKeyCondition.

https://github.com/guardian/scanamo/blob/9cb47068bc3f69709c5bdb1a792dd9a6d5562e01/src/main/scala/com/gu/scanamo/query/DynamoKeyCondition.scala#L26

http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_Condition.html

Table does not typecheck in presentation compiler

I get red squigglies with this sort of code

  val table = Table[MyItem]("my")

a workaround to get typechecking in the rest of the file is to add the type parameter

  val table: Table[MyItem] = Table[MyItem]("my")

(still a red squiggly in the RHS).

I suspect there is a bug in the macro, to add a unit test for this is quite simple, see https://github.com/ensime/pcplod I'm happy to help you get to the bottom of it.

Add support for conditional writes

Currently conditional put and delete are not supported. They probably should be.

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.SpecifyingConditions.html

'Table#getAll' should handle empty 'KeyList' gracefully

Currently Table#getAll would result in com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The list of keys in RequestItems for BatchGetItem is required: my_table has empty list (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: ...) if the KeyList or MultipleKeyList is empty.

It would be better if it silently returns an empty list so you don't have to perform the check before calling Table#getAll

Support to send custom HTTP headers

Currently the ScanamoAsync functions does not provide any option to set custom HTTP headers. It would be nice to support an equivalent function putCustomRequestHeader from aws-java-sdk-dynamodb.

How to limit results size when using query

Is there a way to limit the result size using ScanamoAsync.query?

Thanks in advance!

Cannot set a field to `None` when doing an item update

    table.update(
      'id -> recipe.id,
      set('bakeSchedule -> bakeSchedule) // bakeSchedule == None
    )

fails with:

com.amazonaws.AmazonServiceException: Invalid UpdateExpression: An expression attribute value used in expression is not defined; attribute value: :r_update (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: PMVV1B4SD4DPG6KBAOUU2JJDPVVV4KQNSO5AEMVJF66Q9ASUAAJG)
        at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1370)
        at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:917)
        at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:695)
        at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:447)
        at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:409)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:358)
        at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2051)
        at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:2021)
        at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1848)
        at com.gu.scanamo.ops.ScanamoInterpreters$$anon$1.apply(ScanamoInterpreters.scala:74)
        at com.gu.scanamo.ops.ScanamoInterpreters$$anon$1.apply(ScanamoInterpreters.scala:49)

Aside: If Scanamo had support for REMOVE then I could work around this like so:

    table.update(
      'id -> recipe.id,
      if (bakeSchedule.isDefined) set('bakeSchedule -> bakeSchedule) else remove('bakeSchedule)
    )

I'll try and put together a PR to support REMOVE.

Support Cats 1.0.0

Scanamo should support cats 1.0.0.

List scanamo in cats readme?

May I list scanamo in cats readme as a related project in the cats ecosystem section? I am collecting useful libraries for new users to start applications in FP.

moreHelpfulErrorMessage sometimes not so helpful

Just ran into this in the wild—if a higher priority instance fails but a lower priority one doesn't, there's still a message that looks like a compilation failure:

scala> import com.gu.scanamo.DynamoFormat
import com.gu.scanamo.DynamoFormat

scala> class Foo(x: Int); class Bar(y: Int)
defined class Foo
defined class Bar

scala> trait Low { implicit val fooFormat: DynamoFormat[Foo] = ??? }
defined trait Low

scala> object High extends Low {
     |   implicit def anotherFooFormat(implicit bf: DynamoFormat[Bar]): DynamoFormat[Foo] = ???
     | }
defined object High

scala> import High._
import High._

scala> def getFooFormat = DynamoFormat[Foo]
<console>:19: could not find implicit value for parameter instance: com.gu.scanamo.DynamoFormat[Bar]
       def getFooFormat = DynamoFormat[Foo]
                                      ^
getFooFormat: com.gu.scanamo.DynamoFormat[Foo]

This can be pretty confusing when you're scrolling through a bunch of compilation errors and a few are actually [info] and not errors at all.

Lack of documentation for conditional expressions on put and delete

The code is definitely there (e.g. PR #31), but I have not yet figured out how to use it. Some documentation would be great, or at the very least a brief example on the PR or in a test.

Support assigning attributes to other attribute values during SetExpression

Using the Java client you are able to set attributes to the value of other attributes during an update. The example below sets someField to the value of someOtherField if someField does not exist:

val updateItemSpec = new UpdateItemSpec()
      .withPrimaryKey("someId", someId)
      .withReturnValues(ReturnValue.ALL_NEW)
      .withUpdateExpression("set someField=someOtherField")
      .withConditionExpression(s"attribute_not_exists(someField)")

Is it possible to modify SetExpression to allow this functionality?

Support batch deletion

There is putAll and getAll, would deleteAll make sense as well? This doc also demonstrates a mix of puts and deletes:

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/batch-operation-document-api-java.html

Perhaps there could be syntax for mixing the two like batch(put = List(...), delete = List(...))

Name of the primary key should be flexible

Hi @philwills ,

First of all thank you for sharing this library!

Unfortunately playing with it, I found a bug.

You can create table and use table only if primary key have name name.

Otherwise you will get an error com.amazonaws.AmazonServiceException, with message: One of the required keys was not given a value (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException;

You can easy reproduce this bug by changing, for example, ScanamoAsyncTest "should put asynchronously":

LocalDynamoDB.usingTable(client)("asyncFarmers")('somethingElse -> S) {
      ...
      case class Farmer(somethingElse: String, age: Long, farm: Farm)

Thank you again for the library,
Best Regards,
Pavlo

Should LocalDynamoDB object be in src/main?

The current set up doesn't match the documentation. It can't be used like that, because it's in src/test.
To me it seems like it's inteded place is be src/main.

Unused discipline dependency

Scanamo depends on discipline but doesn't seem to use it. It's also not in test scope. Is it safe to remove, if only for smaller jar size?

Dynamo NULL type handling

We have lot’s of fields that are wrapped in an Option that are currently being stored in Dynamo as a NULL type. Scanamo isn’t able to coerce them to a None.

Support different types for hash key and range key

get, delete and so on take the key as (key: (Symbol, K)*), but this means that all the components of the primary key have to have the same type K. This seems like quite a special case to me.

For example, in my app, the hash key is a RecipeId (with a DynamoFormat to persist it as a string) and the range key is the build number, an Int.

What about just providing 2 versions of each method that requires a key? e.g.

def get[K, T](client: AmazonDynamoDB)(tableName: String)(hashkey: (Symbol, K))
    (implicit fk: DynamoFormat[K], ft: DynamoFormat[T]): Option[ValidatedNel[DynamoReadError, T]] = ...

def get[HK, RK, T](client: AmazonDynamoDB)(tableName: String)(hashkey: (Symbol, HK), rangekey: (Symbol, RK))
    (implicit fhk: DynamoFormat[HK], frk: DynamoFormat[RK], ft: DynamoFormat[T]): Option[ValidatedNel[DynamoReadError, T]] = ...

The varargs param strikes me as a bit odd anyway, as you will only ever pass exactly 1 or 2 keys.

I can put a PR together if you're happy with this suggestion.

LocalDynamoDB.client() is not working

What do I do to get connected to my dynamodb not local?

I made a client like below but it's still not working.. please help..

val client = AmazonDynamoDBClientBuilder.standard()
.withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("dynamodb.ap-northeast-2.amazonaws.com", "ap-northeast-2"))
.build()

How do you build a dynamic update query?

Hello everyone,

I was wondering how to build a dynamic update instruction as I have a use case where I want to update one or more fields in DynamoDB and did not find a clean way.

I have an update coming in that looks like this:

case class IncomingUpdate(firstName: Option[String], xCurrency: Option[Int], yCurrency: Option[Int])

Not all fields could be present and these map onto DynamoDB. I was wondering how you could build an update query in a nice manner.

I detailed my attempts at trying to come up with something over here but alas with no luck:
https://gitter.im/guardian/scanamo?at=57bdd2bb5bdd197c1cb9e405

getAll requires manual batching

Currently putAll operations work around the maximum batch size limitation of DynamoDB by batching the operations behind the scenes, but inconsistently getAll requires the user to do the batching themselves.

getAll should manage the batching so that the user of the library doesn't have to worry about it.

Update to latest aws sdk

Hi,

Are you able to please update this to the latest aws sdk version?

Thanks!

How to get a DynamoFormat[T] from a external case class?

If I define a case class in the scope of my Scanamo usage, I don't have a problem.

If I import a case class from elsewhere, the compiler isn't able to find one.

Reading the code, I see Also supports automatic derivation for case classes in the scaladocs for DynamoFormat[T] and I know it works in the first scenario. I just don't know how to bring the custom format into scope.

error: could not find implicit value for evidence parameter of type com.gu.scanamo.DynamoFormat[com.compay.readmodel.Model]

Support for nested fields in conditional expressions

Currently Scanamo supports conditions only on top-level item fields. It would be great to be able to use nested fields in conditional expressions similar to nested field updates.

I've reused Field definition from nested updates and put up a small proof of concept for the simplest equality testing use-case:

object NestedFieldTest {
  case class Record(id: String, nested: Nested)
  case class Nested(s: String, i: Int)
}

class NestedFieldTest extends FunSpec with Matchers {
  import NestedFieldTest._

  lazy val client = AmazonDynamoDBClientBuilder.standard().withEndpointConfiguration(new EndpointConfiguration("http://localhost:8000", "")).build()
  lazy val table = Table[Record]("TestTable")

  implicit def fieldValueEqualsCondition[V: DynamoFormat] = new ConditionExpression[(Field, V)] {
    override def apply(pair: (Field, V))(condition: Option[RequestCondition]): RequestCondition =
      RequestCondition(
        s"#${pair._1.placeholder} = :condition",
        pair._1.attributeNames.map { case (k, v) => s"#$k" -> v },
        Some(Map(s":condition" -> DynamoFormat[V].write(pair._2)))
      )
  }

  describe("Scanamo") {
    val id = "ABC"
    val putRequest = table.put(Record(id, Nested("A", 0)))
    Scanamo.exec(client)(putRequest)

    it("should allow to use nested field in condition expressions") {
      val updateRequest = table
        .given('nested \ 's -> "A")
        .update('id -> id, set('nested \ 's -> "B") and set('nested \ 'i -> 42))
      val updated = Scanamo.exec(client)(updateRequest)
      updated.right.value should equal(Record(id, Nested("B", 42)))
    }

    it("should combine condition expressions with nested fields") {
      val updateRequest = table
        .given(Condition('nested \ 's -> "B") and ('nested \ 'i -> 42))
        .update('id -> id, set('nested \ 'i -> 1000))
      val updated = Scanamo.exec(client)(updateRequest)
      updated.right.value should equal(Record(id, Nested("B", 1000)))
    }
  }
}

This will probably fail for more complex scenarios, such as nested name conflicts (e.g. 'a \ 'c and 'b \ 'c in one combined expression), but should be not hard to fix and extend to other conditions.

Support UpdateItem

Support for partial updates would be handy

querying with two conditions

Hi --

I would like to write a query with two equality conditions. Yet while example works:

  'mode -> "Underground" and ('line beginsWith "C")

but this does not compile:

 'mode -> "Underground" and ('line -> "C")

what is the reason behind this?

Filter query/scan results

Is there a way to apply a filter to query/scan results?

Cannot update a nested attribute

Prior to the 0.9.0 release, we were providing our own UpdateExpression typeclass implementation (NestedSetExpression) to handle this. With 0.9.x, UpdateExpression was changed from a typeclass-based approach to a sealed trait hierarchy, so we can no longer provide our own implementation.

Is there a way to handle this in 0.9 that I'm missing? If not, what would be the preferred approach? Is it possible to have the existing SetExpression handle nested fields, i.e. set("myPage.isViewed", true)? Or would it be better to have a separate implementation, as we did (i.e. something like setN instead of set)?

Pattern for supporting other collection types

I attempted to define a DynamoFormat for scala's Vector, using listFormat as a template and found that the some useful helper functions (attribute, javaListFormat) are marked private. It would be convenient have access to these and avoid duplicating them in client code, or have an alternative helper for users that want to customize collection serialization.

Support UpdateItemRequest.withReturnValues

When updating an item, it's possible to specify the values you want returned:

• NONE (default)
• ALL_OLD
• UPDATED_OLD
• ALL_NEW
• UPDATED_NEW

I need access to the updated value here, but Scanamo doesn't currently support this.

I can have a go at a PR, but I'm not sure whether to just add an argument (with a default param) to Table.update or to try and add something to the query DSL. I don't see any other default args in the Scanamo API; I'm not sure how @philwills feels about them.

Add option for consistent read on Get operation

In using scanamo the past few months, I came across an opportunity for a possible feature addition. My team currently has a need to do a consistent read on a Get operation from DynamoDB. Is this option a feature that could be added/accepted into the scanamo library?

I've put together a PR that is my first pass at adding this in for a Get operation. The idea is that the consistent read option would default to false, causing it to work as it does today, but a user could optionally pass in a value of true if a consistent read was needed.

Feedback and/or suggestions are welcome! Thanks!

Query in reverse order

QueryRequest and QuerySpec have a withScanIndexForward method. Is there existing dsl syntax to reverse the query order?

Add "attribute_not_exists" condition expression

Currently only "attribute_exists" is supported.
Add "attribute_not_exists" condition expression to select only items with a missing attribute.

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Expressions.ConditionExpressions.html

java.lang.NoSuchMethodError Exception

I'm getting the following exception:

java.lang.NoSuchMethodError: com.amazonaws.services.dynamodbv2.model.AttributeValue.withM

Here is what I am trying to do:

import com.amazonaws.auth.BasicAWSCredentials
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient
import com.gu.scanamo._
import com.gu.scanamo.syntax._
import com.amazonaws.services.dynamodbv2.model.ScalarAttributeType._

object Test {
  case class People(name: String)

  def getPerson = {
    val s = new BasicAWSCredentials("accessKey", "secretKey")
    val client = new AmazonDynamoDBClient(s)
    client.setEndpoint("http://dynamodb.us-west-2.amazonaws.com")

    println(Scanamo.get[People](client)("People")('name -> "David"))
  }
}

Create query with hash equals and range equals

Hi,
Looking for an example of creating a query with both has equals and range equals. There is one with beginsWith but not equals. Also how to add attribute expressions to query?

Thanks in advance
Keshav

Examples of how to interact with new/pre-existing tables with Hash and Sort keys

It would be nice to have some more examples showing of this behavior 😄