rudolph-miller/dyna

Dyna is an AWS DynamoDB ORM for Common Lisp.

Dyna

Dyna is an AWS DynamoDB ORM for Common Lisp.

Usage

(defvar *dyna* (make-dyna :credentials (cons (asdf::getenv "AWS_ACCESS_KEY")
                                             (asdf::getenv "AWS_SECRET_KEY"))
                 :region "ap-northeast-1"))
 
(defclass thread ()
  ((forum-name :key-type :hash
               :attr-name "ForumName"
               :initarg :forum-name
               :accessor thread-forum-name)
   (subject :key-type :range
            :attr-name "Subject"
            :initarg :subject
            :accessor thread-subject))
  (:dyna *dyna*)
  (:table-name "Thread")
  (:metaclass <dyna-table-class>))

(migrate-dyna-teble 'thread)
;; => T

(save-dyna (make-instance 'thread :forum-name "Amazon DynamoDB"
                                  :subject "Really useful"))
;; => T

(find-dyna 'thread "Amazon DynamoDB" "Really useful")
;; => #<THREAD :forum-name "Amazon DynamoDB" :subject "Really useful">



;;; The operations below is the samples of Low Level API.

(fetch (dyna-credentials *dyna*) "local" "ListTables" "{}")
;; => #(...)

(put-item *dyna* :table-name "aliens"
                 :item (("Name" . "LispAlien") ("Feature" . "They talk Lisp.")))
;; => T

(get-item *dyna* :table-name "aliens" :key (("Name" . "LispAlien"))))
;; => (("Name" . "LispAlien") ("Feature" . "They talk Lisp."))

Installasion

(ql:quickload :dyna)

API

dyna

(make-dyna :credentials (cons "access-key" "secret-key")
           :region "ap-northeast-1")

• dyna object is a object for setting up.
• make-dyna creates dyna object.
• :credentials is a dotted pair of AccessKey and SecretKey.
• :region is a region of your DynamoDB.
• If you want to access you local DynamoDB Local,
  you can setup :region "local" and (setf *local-port* 8000).

(defclass thread ()
  ((forum-name :key-type :hash
               :attr-name "ForumName"
               :attr-type :S
               :initarg :forum-name
               :accessor thread-forum-name)
   (subject :key-type :range
            :attr-name "Subject"
            :attr-type :S
            :initarg :subject
            :accessor thread-subject)
   (owner :attr-name "Owner"
          :attr-type :S
          :initarg :owner
          :accessor thread-owner)
   (last-post-date-time :attr-name "LastPostDateTime"
                        :attr-type :S
                        :initarg :last-post-date-time
                        :accessor thread-last-post-date-time))
  (:dyna *dyna*)
  (:table-name "Thread")
  (:throughput (:read 1 :wirte 1)
  (:lsi lat-post-date-time)
  (:gsi (:hash owner :read 5 :write 5))
  (:metaclass <dyna-table-class>))

;; Simpler Style

(defclass thread ()
  ((forum-name :key-type :hash
               :attr-type :S
               :initarg :forum-name
               :accessor thread-forum-name)
   (subject :key-type :range
            :attr-type :S
            :initarg :subject
            :accessor thread-subject))
  (:dyna *dyna*)
  (:metaclass <dyna-table-class>))

• You can create class haveing as :metaclass.
• :dyna can take dyna object.
• :table-name can take table name of DynamoDB's table. (Optional)
• :throughput is the ProvisionedThroughput of the table. (Optional)
• You can create table without :throughput,
  then the first value of ProvisionedThroughput will be *default-throughput*,
  and you can adjust ProvisionedThroughput with AWS Console.
• :lsi is columns of LocalSecondaryIndexes.
• :gsi is columns of GlobalSecondaryIndexes.
• You can create :gsi without :read nor :write,
  then the first value of ProvisionedThroughput in GlobalSecondaryIndexes will be
  *default-throughput*, and you can adjust ProvisionedThroughput with AWS Console.
• :key-type in slot should be :hash or :range and is the same as DynamoDB's table.
• :attr-name in slot is AttributeName of Item in DynamoDB's table. (Optional)
• :attr-type in slot is AttributeType of Item in DynamoDB's table. (Optional)
• You must attach :attr-type with Attributes used in Indexes.

create-dyna-table

(create-dyna-table 'thread)
;; => T

• can return T if the table is successfully created.

update-dyna-table

(defclass thread ()
  ((forum-name :key-type :hash
               :attr-name "ForumName"
               :attr-type :S
               :initarg :forum-name
               :accessor thread-forum-name)
   (subject :key-type :range
            :attr-name "Subject"
            :attr-type :S
            :initarg :subject
            :accessor thread-subject))
  (:dyna *dyna*)
  (:table-name "Thread")
  (:throughput (:read 10 :wirte 10)
  (:metaclass <dyna-table-class>))
(update-dyna-table 'thread)
;; => T

(update-dyna-table 'thread)
;; => NIL

• can return T if the table is successfully updated.
• can return NIL if the table has no changs.

migrate-dyna-table

(migrate-dyna-table 'thread)
=> T

• can create the table if the table doesn't exist.
• can update the table if the table definitions are chagned.
• can return NIL if the table has no changes.

find-dyna

(find-dyna 'thread "Amazon DynamoDB" "Really useful")
;; => #<THREAD :forum-name "Amazon DynamoDB" :subject "Really useful">

• can return a object if matching Item exists.

select-dyna

(select-dyna 'thread)
;; => (#<THREAD > <#THREAD >)

(selet-dyna 'thread (where (:= :forum-name "Amazon DynamoDB")))
;; => (#<THREAD >)

(selet-dyna 'thread (where (:or (:= :forum-name "Amazon S3")
                                (:= :forum-name "Amazon DynamoDB"))))
;; => (#<THREAD > #<THREAD >)

(selet-dyna 'thread (where (:in :forum-name '("Amazon S3" "Amazon DynamoDB"))))
;; => (#<THREAD > #<THREAD >)

(selet-dyna 'thread (where (:in :forum-name '("Amazon S3" "Amazon DynamoDB")))
                    :limit 1)
;; => (#<THREAD >)

(selet-dyna 'thread (where (:in :forum-name '("Amazon S3" "Amazon DynamoDB")))
                    (limit 1))
;; => (#<THREAD >)

(select-dyna 'thread :segments 4)
;; => (#<THREAD > <#THREAD >)

• returns the list of objects.
• can handle Extended Where Clause of SxQL.
• can handle LastEvaluatedKey in the response.
• :limit can restrict the number of results.
• can handle Limit Clause of SxQL.
• :segments can make scan request divided.

save-dyna

(save-dyna (make-instance 'thread :forum-name "Amazon DynamoDB"
                                  :subject "Really useful"))
;; =>

• can return T if the object is successfully saved.

Low Level API

Most Low Level API return multiple values, the formaer is formatted result, and the latter is raw result.

fetch

(fetch (cons "access-key" "secret-key") "ap-northeast-1" "ListTables" "{}")
;; => #(...)

• returns raw octets of reponse.

batch-get-item

(batch-get-item dyna :request-items '(("Forum" . (("Keys" . ((("Id" . 1))
                                                             (("Id" . 2))))
                                                  ("ProjectionExpression" . "Id, Title, Author")))
                                      ("Thread" . (("Keys" . ((("ForumName" . "Amazon DynamoDB")
                                                               ("Subject" . "Concurrent reads"))))
                                                  ("AttributesToGet" . "ForumName, Subject"))))
                     :return-consumed-capacity "TOTAL")))
;; => (("Forum" (("Id" . 1) ("Title" . "Enjoy Lisp") ("Author" . "Rudolph-Miller"))
;;              (("Id" . 2) ("Title" . "Sophisticated Programming Language") ("Author" . "Lisp-Alien")))
;;     ("Thread" (("ForumName" . "Amazon DynamoDB") ("Subject" . "Concurrent reads"))))

• returns the list of alists.
• Support
  • :request-items
  • :return-consumed-capacity

batch-write-item

(batch-write-item dyna :request-items '(("Forum" . ((("PutRequest" . (("Item" . (("Name" . "Amazon DynamoDB")
                                                                                 ("Category" . "Amazon Web Services"))))))
                                                    (("PutRequest" . (("Item" . (("Name" . "Amazon RDS")
                                                                                 ("Category" . "Amazon Web Services"))))))))
                                        ("Thread" . ((("PutRequest" . (("Item" . (("ForumName" . "Amazon DynamoDB")
                                                                                  ("Subject" . "Concurrent reads")))))))))
                       :return-consumed-capacity "TOTAL")
;; => T

• returns t if the operation succeeded.
• Support
  • :request-items
  • :return-consumed-capacity
  • :return-item-collection-metrics

create-table

(create-table dyna :table-name "Thread"
                   :key-schema '((("AttributeName" . "ForumName") ("KeyType" . "HASH"))
                                 (("AttributeName" . "Subject") ("KeyType" . "RANGE")))
                   :attribute-definitions '((("AttributeName" . "ForumName") ("AttributeType" . "S"))
                                            (("AttributeName" . "Subject") ("AttributeType" . "S"))
                                            (("AttributeName" . "LastPostDateTime") ("AttributeType" . "S")))
                   :local-secondary-indexes '((("IndexName" . "LastPostIndex")
                                               ("KeySchema" . ((("AttributeName" . "ForumName")
                                                                ("KeyType" . "HASH"))
                                                               (("AttributeName" . "LastPostDateTime")
                                                                ("KeyType" . "RANGE"))))
                                               ("Projection" . (("ProjectionType" . "KEYS_ONLY")))))
                   :provisioned-throughput '(("ReadCapacityUnits" . 5)
                                             ("WriteCapacityUnits" . 5)))
;; => T

• returns t if the operation succeeded.
• Support
  • :table-name
  • :attribute-definitions
  • :key-schema
  • :global-secondary-indexes
  • :local-secondary-indexes
  • :provisioned-throughput

delete-item

(delete-item dyna :table-name "Thread"
                  :key '(("ForumName" . "Amazon DynamoDB"))
                  :condition-expression "attribute_not_exists(Replies)"
                  :return-values "ALL_OLD")
;; => T

• returns t if the operation succeeded.
• Support
  • :table-name
  • :key
  • :condition-expression
  • :return-values
  • :expression-attribute-names
  • :expression-attribute-values
  • :return-consumed-capacity
  • :return-item-collection-metrics

delete-table

(delete-table dyna :table-name "Thread")
;; => T

• returns t if the operation succeeded.
• Support
  • :table-name

describe-table

(describe-table dyna :table-name "Thread")

• returns the jsown object of table description.
• Support
  • :table-name

get-item

(get-item dyna :table-name "Thread"
               :key '(("Tags" . ("Multiple Items" "HelpMe")))
               :consistent-read t
               :return-consumed-capacity "TOTAL")
;; => (("Tags" "Multiple Items" "HelpMe") ("ForumName" . "Amazon DynamoDB"))

• returns the alist of item.
• Support
  • :table-name
  • :key
  • :projection-expression
  • :consistent-read
  • :return-consumed-capacity
  • :expression-attribute-names

list-tables

(list-tables-content dyna)
;; => ("Thread")

• returns the list of table names.
• Support
  • :exclusive-start-table-name
  • :limit

put-item

(put-item dyna :table-name "Thread"
                :item '(("Tags" . ("Multiple Items" "HelpMe"))
                        ("ForumName" . "Amazon DynamoDB"))
                :condition-expression "ForumName <> :f and Subject <> :s"
                :expression-attribute-values '((":f" . "Amazon DynamoDB")
                                               (":s" . "update multiple items")))
;; => T

• returns t if the operation succeeded.
• Support
  • :table-name
  • :item
  • :on-expression
  • :expression-attribute-values
  • :expression-attribute-names
  • :return-consumed-capacity
  • :return-item-collection-metrics
  • :return-values

query

(query dyna
       :table-name "Thread"
       :select "SPECIFIC_ATTRIBUTES"
       :attributes-to-get '("ForumName" "Subject")
       :limit 3
       :key-conditions '(("ForumName" . (("AttributeValueList" . ("Amazon DynamoDB"))
                                         ("ComparisonOperator" . "EQ"))))
       :return-consumed-capacity "TOTAL")
;; => ((("ForunName" . "Amazon DynamoDB") ("Subject" . "Concurrent reads")))

• returns list of alists.
• Support
  • :table-name
  • :key-conditions
  • :return-consumed-capacity
  • :attributes-to-get
  • :index-name
  • :select
  • :limit
  • :consistent-read
  • :conditional-operator
  • :exclusive-start-key
  • :expression-attribute-values
  • :expression-attribute-names
  • :filter-expression
  • :projection-expression
  • :query-filter
  • :scan-index-forward

scan

(scan dyna :table-name "Thread"
           :projection-expression "ForumName,Subject"
           :limit 3
           :filter-expression "Replies > :num"
           :expression-attribute-values '((":num" . 10))
           :scan-index-forward t
           :return-consumed-capacity "TOTAL")
;; => ((("ForunName" . "Amazon DynamoDB") ("Subject" . "Concurrent reads")))

• returns list of alists.
• Support
  • :table-name
  • :key-conditions
  • :return-consumed-capacity
  • :attributes-to-get
  • :index-name
  • :select
  • :limit
  • :consistent-read
  • :conditional-operator
  • :exclusive-start-key
  • :expression-attribute-values
  • :expression-attribute-names
  • :filter-expression
  • :projection-expression
  • :scan-filter
  • :scan-index-forward
  • :segment
  • :total-segments

update-item

(update-item dyna :table-name "Thread"
                  :key '(("ForumName" . "Amazon DynamoDB"))
                  :update-expression "set Replies = Replies + :num"
                  :expression-attribute-values '((":num" . 1))
                  :return-values "NONE")
;; => T

• returns t if the operation succeeded.
• Support
  • :table-name
  • :key
  • :update-expression
  • :expression-attribute-names
  • :expression-attribute-values
  • :return-values
  • :return-consumed-capacity
  • :return-item-collection-metrics

update-table

(update-table dyna :table-name "Thread"
                   :attribute-definitions '((("AttributeName" . "ForumName")
                                            ("AttributeType" . "S"))
                                            (("AttributeName" . "Subject")
                                            ("AttributeType" . "S"))
                                            (("AttributeName" . "LastPostDateTime")
                                            ("AttributeType" . "S")))
                   :provisioned-throughput '(("ReadCapacityUnits" . 5)
                                             ("WriteCapacityUnits" . 5))
                   :global-secondary-index-updates '((("Create" . (("IndexName" . "LastPostIndex")
                                                                   ("KeySchema" . ((("AttributeName" . "Subject")
                                                                                    ("KeyType" . "HASH"))
                                                                                   (("AttributeName" . "LastPostDateTime")
                                                                                    ("KeyType" . "RANGE"))))
                                                                   ("Projection" .(("ProjectionType" . "ALL")))
                                                                   ("ProvisionedThroughput" . (("ReadCapacityUnits" . 5)
                                                                                    ("WriteCapacityUnits" . 5))))))))
;; => T

• returns t if the operation succeeded.
• Support
  • :table-name
  • :attribute-definitions
  • :provisioned-throughput
  • :global-secondary-index-updates

Extended Where Clause

• You can use ordinary operators.
• Extended operators are below. (:between, :begins-with, :contains, :list=, :list-in)

(where (:between :forum-name '("A" "Z")))
(where (:begins-with :forum-name "Amazon"))
(where (:contains :forum-name "RDS"))

;; If Attribute Type of "Tags" is "SS", you query := and :in with :list= and :list-in respectively.
(where (:list= :tags '("AWS" "Easy")))
(where (:list-in :tags '(("AWS" "Easy") ("Scalable"))))

See Also

• SxQL - A SQL generator.

Author

• Rudolph-Miller ([email protected])

Copyright

Copyright (c) 2015 Rudolph-Miller ([email protected])

License

Licensed under the MIT License.

dyna's Issues

examples

update-dyna

projection with 予約語

http://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/developerguide/ReservedWords.html

contains

dyna-table object

use jonathan

handle throughput error

find-dyna

increment

use local secondary index

error with changing lsi

where clause

custom header on http request

support unsupported params

:= with set or list

exceeded throughput

handle or not

Some systems failed to build for Quicklisp dist

Building with SBCL 2.0.9.125-5f9d19d4a / ASDF 3.3.1 for quicklisp dist creation.

Commit id d8cf27b

dyna-test fails to build with the following error:

Unhandled SB-EXT:PACKAGE-LOCKED-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {100E548103}>: Lock on package SXQL.OPERATOR violated when importing symbol LIST=-OP while in package DYNA.SXQL.

dyna fails to build with the following error:

Unhandled SB-EXT:PACKAGE-LOCKED-ERROR in thread #<SB-THREAD:THREAD "main thread" RUNNING {100E570103}>: Lock on package SXQL.OPERATOR violated when importing symbol LIST=-OP while in package DYNA.SXQL.

Full log here

:list-in

like :list=

is-null and not-null

use cl-annot-prove and generate document

delete-dyna

bench

global secondary index

migrate list= and list-in

Can't use temporary credentials (no aws session token)

The AWS account I have to access uses security via temporary credentials. There's no way to set the AWS Session Token in Dyna, so all requests return 400.

select-dyna

construct http request

Broken after latest quicklisp update

Current version installed by ql:quickload doesn't work.

Error occured in request.
Status: 400
Messssage: {"__type":"com.amazon.coral.service#InvalidSignatureException","message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}
Meta: (date Mon, 09 Apr 2018 14:32:22 GMT content-length 254
       content-type application/x-amz-json-1.0 x-amz-crc32
       1393778603 x-amzn-requestid).

I suppose that dexador update might be the root cause.
Working request for me (from old setup):

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
POST / HTTP/1.1
User-Agent: Dexador/0.9.10 (SBCL 1.4.5); Linux; 4.15.15-gentoo-ray
Host: dynamodb.eu-west-1.amazonaws.com
Accept: */*
Content-Type: application/x-amz-json-1.0
Content-Length: 2
Authorization: AWS4-HMAC-SHA256 Credential=HERE_WAS_MY_AWS_ACCESS_KEY/20180409/eu-west-1/dynamodb/aws4_request, SignedHeaders=host;content-type;x-amz-date;x-amz-target, Signature=5efe6ee55f46d7abac35f64f6cea07be8dc08973721269e4a7234f54613027e7
X-Amz-Date: 20180409T140535Z
X-Amz-Target: DynamoDB_20120810.ListTables

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
HTTP/1.1 200 OK
x-amzn-RequestId: DF2PCSS4UV8OV21873VRT3O7ARVV4KQNSO5AEMVJF66Q9ASUAAJG
x-amz-crc32: 421541087
Content-Type: application/x-amz-json-1.0
Content-Length: 26
Date: Mon, 09 Apr 2018 14:05:34 GMT

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Broken request (latest quicklisp version):

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
POST / HTTP/1.1
User-Agent: Dexador/0.9.10 (SBCL 1.4.5.debian); Linux; 4.15.15-gentoo-ray
Host: dynamodb.eu-west-1.amazonaws.com
Accept: */*
Content-Type: application/x-amz-json-1.0
Authorization: AWS4-HMAC-SHA256 Credential=HERE_WAS_MY_AWS_ACCESS_KEY_AGAIN/20180409/eu-west-1/dynamodb/aws4_request, SignedHeaders=host;content-type;x-amz-date;x-amz-target, Signature=555c26ea2f84e6aff8b0fd20c0bbc3c9780cb430ac40ddb5335e56645041fc73
X-Amz-Date: 20180409T140058Z
X-Amz-Target: DynamoDB_20120810.ListTables

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
HTTP/1.1 400 Bad Request
x-amzn-RequestId: CL3MJGR0494FORGUOD8685NB2NVV4KQNSO5AEMVJF66Q9ASUAAJG
x-amz-crc32: 1393778603
Content-Type: application/x-amz-json-1.0
Content-Length: 254
Date: Mon, 09 Apr 2018 14:00:58 GMT

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

At least Content-Length field disappeared.

handler order-by

scan with hash-key => query with filter

limit clause

Authorization

Name conflict - does not build on sbcl

http://report.quicklisp.org/2016-11-08/failure-report/dyna.html#dyna has a log.

Unhandled SB-EXT:NAME-CONFLICT in thread #<SB-THREAD:THREAD "main thread" RUNNING {1002D4F5C3}>: IMPORT LIST=-OP causes name-conflicts in #<PACKAGE "SXQL.OPERATOR"> between the following symbols:DYNA.SXQL::LIST=-OP, SXQL.OPERATOR::LIST=-OP

auto migration

BEGINS_WITH

use dexador

parallel scan

ignore throughput without specified

api

between