jeremydaly/dynamodb-toolbox

A simple set of tools for working with Amazon DynamoDB and the DocumentClient

DynamoDB Toolbox

Single Table Designs have never been this easy!

The DynamoDB Toolbox is a set of tools that makes it easy to work with Amazon DynamoDB and the DocumentClient. It's designed with Single Tables in mind, but works just as well with multiple tables. It lets you define your Entities (with typings and aliases) and map them to your DynamoDB tables. You can then generate the API parameters to put, get, delete, update, query, scan, batchGet, and batchWrite data by passing in JavaScript objects. The DynamoDB Toolbox will map aliases, validate and coerce types, and even write complex UpdateExpressions for you. 😉

Why single table design?

Learn more about single table design in Alex Debrie's blog.

Version 0.9 🙌

Feedback is welcome and much appreciated! (Huge thanks to @ThomasAribart for all his work on this 🙌)

Using AWS SDK v2?

[email protected] and above is using AWS SDK v3. If you are using AWS SDK v2, please use versions that are lower than 0.8.

Docs & Community

• Full Documentation
• GitHub Discussions

Quick Start

Using your favorite package manager, install DynamoDB Toolbox and the aws-sdk v3 dependencies in your project by running one of the following commands:

# npm
npm i dynamodb-toolbox
npm install @aws-sdk/lib-dynamodb @aws-sdk/client-dynamodb

# yarn
yarn add dynamodb-toolbox
yarn add @aws-sdk/lib-dynamodb @aws-sdk/client-dynamodb

Require or import Table and Entity from dynamodb-toolbox:

import { Table, Entity } from 'dynamodb-toolbox'

Create a Table (with the DocumentClient using aws-sdk v3):

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'
import { DynamoDBDocumentClient } from '@aws-sdk/lib-dynamodb'

const marshallOptions = {
  // Whether to automatically convert empty strings, blobs, and sets to `null`.
  convertEmptyValues: false, // if not false explicitly, we set it to true.
  // Whether to remove undefined values while marshalling.
  removeUndefinedValues: false, // false, by default.
  // Whether to convert typeof object to map attribute.
  convertClassInstanceToMap: false, // false, by default.
}

const unmarshallOptions = {
  // Whether to return numbers as a string instead of converting them to native JavaScript numbers.
  // NOTE: this is required to be true in order to use the bigint data type.
  wrapNumbers: false, // false, by default.
}

const translateConfig = { marshallOptions, unmarshallOptions }

// Instantiate a DocumentClient
export const DocumentClient = DynamoDBDocumentClient.from(new DynamoDBClient({}), translateConfig)


// Instantiate a table
const MyTable = new Table({
  // Specify table name (used by DynamoDB)
  name: 'my-table',

  // Define partition and sort keys
  partitionKey: 'pk',
  sortKey: 'sk',

  // Add the DocumentClient
  DocumentClient
})

Create an Entity:

const Customer = new Entity({
  // Specify entity name
  name: 'Customer',

  // Define attributes
  attributes: {
    id: { partitionKey: true }, // flag as partitionKey
    sk: { hidden: true, sortKey: true }, // flag as sortKey and mark hidden
    age: { type: 'number' }, // set the attribute type
    name: { type: 'string', map: 'data' }, // map 'name' to table attribute 'data'
    emailVerified: { type: 'boolean', required: true }, // specify attribute as required
    co: { alias: 'company' }, // alias table attribute 'co' to 'company'
    status: ['sk', 0], // composite key mapping
    date_added: ['sk', 1] // composite key mapping
  },

  // Assign it to our table
  table: MyTable

  // In Typescript, the "as const" statement is needed for type inference
} as const)

Put an item:

// Create an item (using table attribute names or aliases)
const customer = {
  id: 123,
  age: 35,
  name: 'Jane Smith',
  emailVerified: true,
  company: 'ACME',
  status: 'active',
  date_added: '2020-04-24'
}

// Use the 'put' method of Customer:
await Customer.put(customer)

The item will be saved to DynamoDB like this:

{
  "pk": 123,
  "sk": "active#2020-04-24",
  "age": 35,
  "data": "Jane Smith",
  "emailVerified": true,
  "co": "ACME",
  // Attributes auto-generated by DynamoDB-Toolbox
  "_et": "customer", // Entity name (required for parsing)
  "_ct": "2021-01-01T00:00:00.000Z", // Item creation date (optional)
  "_md": "2021-01-01T00:00:00.000Z" // Item last modification date (optional)
}

You can then get the data:

// Specify primary key
const primaryKey = {
  id: 123,
  status: 'active',
  date_added: '2020-04-24'
}

// Use the 'get' method of Customer
const response = await Customer.get(primaryKey)

Since v0.4, the method inputs, options and response types are inferred from the Entity definition:

await Customer.put({
  id: 123,
  // ❌ Sort key is required ("sk" or both "status" and "date_added")
  age: 35,
  name: ['Jane', 'Smith'], // ❌ name should be a string
  emailVerified: undefined, // ❌ attribute is marked as required
  company: 'ACME'
})

const { Item: customer } = await Customer.get({
  id: 123,
  status: 'active',
  date_added: '2020-04-24' // ✅ Valid primary key
})
type Customer = typeof customer
// 🙌 Type is equal to:
type ExpectedCustomer =
  | {
      id: any
      age?: number | undefined
      name?: string | undefined
      emailVerified: boolean
      company?: any
      status: any
      date_added: any
      entity: string
      created: string
      modified: string
    }
  | undefined

See Type Inference in the documentation for more details.

Features

• Table Schemas and DynamoDB Typings: Define your Table and Entity data models using a simple JavaScript object structure, assign DynamoDB data types, and optionally set defaults.
• Magic UpdateExpressions: Writing complex UpdateExpression strings is a major pain, especially if the input data changes the underlying clauses or requires dynamic (or nested) attributes. This library handles everything from simple SET clauses, to complex list and set manipulations, to defaulting values with smartly applied if_not_exists() to avoid overwriting data.
• Bidirectional Mapping and Aliasing: When building a single table design, you can define multiple entities that map to the same table. Each entity can reuse fields (like pk andsk) and map them to different aliases depending on the item type. Your data is automatically mapped correctly when reading and writing data.
• Composite Key Generation and Field Mapping: Doing some fancy data modeling with composite keys? Like setting your sortKey to [country]#[region]#[state]#[county]#[city]#[neighborhood] model hierarchies? DynamoDB Toolbox lets you map data to these composite keys which will both autogenerate the value and parse them into fields for you.
• Type Coercion and Validation: Automatically coerce values to strings, numbers and booleans to ensure consistent data types in your DynamoDB tables. Validate list, map, and set types against your data. Oh yeah, and sets are automatically handled for you. 😉
• Powerful Query Builder: Specify a partitionKey, and then easily configure your sortKey conditions, filters, and attribute projections to query your primary or secondary indexes. This library can even handle pagination with a simple .next() method.
• Simple Table Scans: Scan through your table or secondary indexes and add filters, projections, parallel scans and more. And don't forget the pagination support with .next().
• Filter and Condition Expression Builder: Build complex Filter and Condition expressions using a standardized array and object notation. No more appending strings!
• Projection Builder: Specify which attributes and paths should be returned for each entity type, and automatically filter the results.
• Secondary Index Support: Map your secondary indexes (GSIs and LSIs) to your table, and dynamically link your entity attributes.
• Batch Operations: Full support for batch operations with a simpler interface to work with multiple entities and tables.
• Transactions: Full support for transaction with a simpler interface to work with multiple entities and tables.
• Default Value Dependency Graphs: Create dynamic attribute defaults by chaining other dynamic attribute defaults together.
• TypeScript Support: v0.4 of this library provides strong typing support AND type inference 😍. Inferred type can still overriden with Overlays. Some Utility Types are also exposed. Additional work is still required to support schema validation & typings.

Additional References

• DocumentClient SDK Reference
• Best Practices for DynamoDB
• DynamoDB, explained.
• The DynamoDB Book

Contributions and Feedback

Contributions, ideas and bug reports are welcome and greatly appreciated. Please add issues for suggestions and bug reports or create a pull request. You can also contact me on Twitter: @jeremy_daly.

dynamodb-toolbox's Issues

Attributes part of composite keys are not returned

I've got an entity with the following attributes:

attributes: {
        pk: {
          type: 'string',
          partitionKey: true,
        },
        sk: {
          sortKey: true,
          hidden: true,
        },
        username: ['sk', 1, { save: true }],
        role: ['sk', 0],
      },

According to the doc, I should get both username and role when querying this entity.
However, I only get the username (which is saved).

The quick fix is easy but I thought I'd report it so that you could fix it or update the doc.

Simplified transaction handling

Transactions using the library are possible by extracting the parameters and assembling the correct structure for the TransactWriteItems and TransactGetItems APIs, but it is a bit bulky. We could expose another class called Transactions that would allow you to simply add an array of Models with the appropriate method call and parameters. The Transactions class could parse the parameters and assemble them in the correct format.

Proposed Implementation

const { Model, Transaction } = require('dynamodb-toolbox')

const MyModel = new Model( ‘MyModel’,  { ... } )
const MyOtherModel = new Model( ‘MyOtherModel’,  { ... } )
const AnotherModel = new Model( ‘AnotherModel’,  { ... } )

let result = await Transaction(
  [
    MyModel.update(someItem1),
    MyModel.update(someItem2),
    MyOtherModel.update(someItem3),
    MyOtherModel.delete(someItem4),
    AnotherMode.put(someItem5)
  ],
  {
    ReturnConsumedCapacity: ‘TOTAL’
  }
)

No need to specify the “write” or “get” transaction type since the library could do this automatically. It could also enforce the method types (delete, update, and put for writes, and get for get) and the maximum of 25 items per (or maybe it could auto split them 🤷‍♂️).

Also would need to allow for ConditionCheck, so that might just need to be passed in the array.

Setting put numeric field to zero throws error

Problem: when creating put parameters that attempt to set a field to zero (0), the Model throws an error that "(field) is a required field".

Expected Result: I expect to be able to set a required field to zero.

Steps to Reproduce: I have a code snippet that reproduces the issue, below.

const { Model } = require('dynamodb-toolbox');

const model = new Model('TestModel', {
  table: 'TestTable',
  partitionKey: 'pk',
  timestamps: true,
  schema: {
    pk: { type: 'string', hidden: true, default: 'Test' },
    count: { type: 'number', required: 'always' },
  },
});

const result = model.put({ count: 0 }); // This throws

Note that I can work around the issue right now by changing the required value from 'always' to false.

Empty ExpressionAttributeValues generated when using conditions

I tried to do

MyEntity.delete(
  {
    pk,
    sk,
  },
  { conditions: { attr: 'username', exists: true } },
)

but I'm getting ExpressionAttributeValues must not be empty because the generated params are

{ TableName: '',
  Key:
   { PK: '',
     SK: '' },
  ExpressionAttributeNames: { '#attr1': 'username' },
  ExpressionAttributeValues: {},
  ConditionExpression: 'attribute_exists(#attr1)' }

I'm not 100% sure my use case is valid DDB but this might be on the library ;)

Add integration tests with dynalite

The current test suite has over 100 tests, but they mostly verify expected parameters. It would be helpful (and safer) to test them against a DynamoDB instance. A cloud version would be best, but since we want TravisCI to run the tests for us, we should be able to use DynamoDB Local.

put fails when a field is required and mapped with map: 'destination'

put fails when field is configured as map: 'field' and required: true

Create an example app

Hi @jeremydaly it would be awesome for newcomers like me to have an example app that shows off how you should structure the code as a best practice. Nothing fancy but at least how you should load table/entities and do some basic operations.

Native JavaScript Date support

DynamoDB doesn't have a date type, but schemas could easily support it by requiring a native JavaScript Date object as an input and converting it to the ISO 8601 format. When parsing raw Items, the library could convert the dates back to a JS Date object (given the fact that this is specified in the schema).

It could potentially support datetime and date since these are both valid ISO 8601 representations.

Wondering if we need other formats, like just time?

Add Table class to wrap Models

As mentioned in #6, in order to achieve multi-model parsing from a single table design, the system needs to be able to understand what Model each Item represents. I was initially thinking about using a "model registry", but I'd rather the concepts translate better. My mental model is this:

A TABLE is directly mapped to a DynamoDB table. It has a KeySchema, AttributeDefinitions, LocalSecondaryIndexes, and GlobalSecondaryIndexes. A table is the raw repository that uses generic names (like pk, sk, and data) to label attributes.

A MODEL is a "schema" definition that is mapped to a Table's Item. A model could represent something more relational, such as a denormalized copy of some data, or it could also represent a part of a larger "entity", such as in the attribute versioning case from Rick Houlihan's recent talk. Models are meant to be flexible in the sense that you are in control of how your data is stored in each Item (or "record").

An ENTITY is any singular, identifiable and separate object. A single Model may represent an Entity, but and Entity may also be made up of multiple Models (refer to the attribute version case linked above). I haven't fully thought through adding an Entity concept to this library, but I could see cases where it might help.

An ACCESS PATTERN is a method used to retrieve Entities and Models from the Table. The NoSQL Workbench for DynamoDB has a concept of "facets", which sort of feels like this, but would require the developer to write the actually retrieval code.

An UPDATE PATTERN is a method used to update Entities and Models in the Table. This may be a simple putItem or updateItem, or it could be a more complex Transaction with multiple updates, condition checks, etc. This would also be the responsibility of the developer to write the implementation.

The first step is adding the higher level wrapper. This would NOT be necessary, as each Model could still be used independently. However, in order to support more sophisticated features, using the Table class would be required.

sort key doesn't support numeric types

If you try to query with a numeric sort key in a GSI, (like PK=x, { lte: 1234 }), you'll get <attr> must be a string

RFC: Composites should be specified on the composite property itself

If I understand correctly, the current method of specifying composite keys occurs with array attribute values. I saw this cause an issue in #32. I believe the solution is to migrate the api to something that looks like this:

Entity({
  attributes: {
    pk: {composite: ['STATIC_WORD', {attr: '_et'}, {attr: 'id'}], delimiter: '#'},
    id: {default: () => nanoid()},
  },
})

This would allow specifying multiple composites, and it would retain the space-saving features of the current implementation. Additionally, I think it's easier to parse. Does this break anything I'm not aware of?

Add schema support for maps

Only top level schemas are supported. By adding support for nested maps, we could enforce types and provide a cleaner way to update data maps. This should be optional, falling back to the current dot notation for unmapped structures.

Proposed implementation

const MyModel = new Model('MyModel',{
  // Specify table name
  table: 'my-dynamodb-table',

  // Define partition and sort keys
  partitionKey: 'pk',
  sortKey: 'sk',

  // Define schema
  schema: {
    pk: { type: 'string', alias: 'id' },
    sk: { type: 'string', hidden: true },
    someMap: {
      type: 'map' ,
      schema: { // $ shorthand
        title: 'string', // support string-based type declarations
        count: { type: 'number', default: 10 }, // support scalars with defaults and other options
        someList: { type: 'list' },
        someNestedMap: { // Supported nested maps with schemas
          type: 'map',
          schema: { ... }
        }
      } 
    }
  }
})

This would enable support for schema validation as well as partial updates.

GitHub Actions + Automated versioned releases to GitHub and NPM

I'm happy to add automated versioned releases to GitHub and NPM via GitHub Actions + Semantic Release if you're interested. I've done this a few times on various projects. It's essentially Continuous Deployment for packages. 😄 You will need to start using https://www.conventionalcommits.org/ though.

v0.2 GSI Entity Indexes

Firstly, thank you for this great library. I'm just getting into NoSQL database design and specifically trying to grasp the concept of a single table design. You probably have this on your roadmap, but I've noticed that you can't yet create an entity that references a partitionKey with a custom GSI string name.

The following Entity setup throws an error Entity requires a partitionKey attribute.

const MyTable = new Table({
    name: "MyTable",
    partitionKey: "pk",
    sortKey: "sk",
    indexes: {
        GSI1: {partitionKey: "sk", sortKey: "data"},
    },
    DocumentClient,
});

const MyEntity = new Entity({
    name: "MyEntity",
    attributes: {
        sk: {partitionKey: "GSI1"}, // This should work I think
        data: {sortKey: "GSI1"},
    },
    table: MyTable,
});

It looks like the keys are correctly set on the Entity but there is a check which only checks the root track.keys.partitionKey here:
https://github.com/jeremydaly/dynamodb-toolbox/blob/v0.2/lib/parseEntityAttributes.js#L29

The keys do exist in the child object under the custom "GSI1" key though track.keys.GSI1.partitionKey.

As I said above, it may just be that this hasn't been implemented yet but just wanted to flag in case it was missed.

Direct invocation of the DocumentClient

When I started building some of these helper utilities, I was only using them to generate specific parameters (like UpdateExpression and ExpressionAttributeNames). Now that additional DynamoDB parameters can be passed directly through each method, automatically calling the appropriate DocumentClient method would avoid the extra step. This could be optional, but the library already requires and uses the DocumentClient to work with set types.

There are use case that would require you to pass in your own version of the DocumentClient. This could include wrapped versions for AWS X-Ray, or perhaps some other observability tool. The client could be passed in on model creation (or as a global config, see #6). We could also support X-Ray out of the box by allowing you to pass in the aws-xray-sdk as a configuration parameter. I need to see how conditionally applying X-Ray would affect tree-shaking algorithms.

Getting "Invalid Table" error

This works locally, but when I deploy to Lambda, the table.constructor.name is empty, causing dynamodb-toolbox to throw https://github.com/jeremydaly/dynamodb-toolbox/blob/v0.2/classes/Entity.js#L46-L69.

I'm using serverless-webpack which may have something to do with it?

Table Configuration Export

Method to export the CloudFormation for your Table configuration to JSON or YAML

Query support

Someone mentioned this to me, but I’m not sure of the correct abstraction level. If the models and types can aid in building queries, then I think this would be useful, but I don’t want to cross into ORM territory.

There are a limited set of sortKey operations, so some simple abstraction could work, but the library probably shouldn’t be parsing queries. There would likely need to be a different way to input it, and I don’t think chaining makes sense in this case (e.g. .field(‘somevalue‘).gt().value(someValue)).

This needs more feedback.

Function default dependency issue

When using a function as a default, if another default relies on it, it stringifies the function instead of calling it. Kind of hard to explain, but here is an example.

attributes: {
        id: {
          partitionKey: true,
          prefix: 'USER_ID#',
          default: () => randomBytes(32).toString('hex'),
        },
        sk: {
          sortKey: true,
          prefix: 'USER_ID#',
          default: (data) => data.pk,
        }
}

The above schema results in the following entity.

{
  pk: "USER_ID#f6591161cfba6267e0e59ca7405164e16733287870d01359ef089b696ae8d826"
  sk: "USER_ID#() => crypto_1.randomBytes(32).toString('hex')"
}

I can easily generate the id before creation and avoid this problem. I just thought it'd be cool to use defaults.

Data conversion functions

These would be like using functions with default, except it would apply every time you input data to the field. This would be useful for enforcing case conversions or other types of formatting (like phone numbers or custom composite keys).

Proposed Implementation

// Define schema
schema: {
  pk: { type: 'string', alias: 'id' },
  sk: { type: 'string', hidden: true },
  someField: { 
    type: 'string', 
    alias: 'name' ,
    convert: (val,data) => { // gets the submitted value and the rest of the values
      return val.toLowerCase() // do something interesting here
    }
  },
}

The data object here should probably NOT be a simple reference so that data in other fields isn't accidentally mutated. Though having a way to alter data in another input might be a useful feature.

v0.2 Get query on secondary index failing due to requiring PK

I think this might just be a lack of my knowledge of DynamoDB again, but I thought that the following test shouldn't throw the error:

'pk' is required

I thought, that because the GSI partitionKey has been specified, that I can run a get query against it without having to include the main table's partitionKey. The error suggests that this is not the case. I'm a bit confused and I'm sure there is a simple answer to this which I'm struggling to find. Any pointer would be greatly appreciated.

const TestTable3 = new Table({
    name: 'test-table',
    partitionKey: 'pk',
    sortKey: 'sk',
    indexes: {
        GSI1: { partitionKey: 'GSI1sk', sortKey: 'data' },
    },
    DocumentClient,
})

const TestEntity3 = new Entity({
    name: 'TestEntity',
    attributes: {
        pk: { type: 'string', partitionKey: true },
        sk: { type: 'string', partitionKey: 'GSI1', sortKey: true },
        data: { type: 'string', sortKey: 'GSI1' },
    },
    table: TestTable3,
})

describe('get', () => {
    it('gets the Key from GSI1 inputs (sync)', async () => {
        const { TableName, Key } = TestEntity3.getParams({
            sk: 'test-sk', // This is the GSI1 partitionKey I also tried the name `GSI1sk`
            data: 'test-data',
        })
        expect(TableName).toBe('test-table')
        expect(Key).toEqual({ sk: 'test-sk', data: 'test-data' })
    })
})

Batch writes does not support conditions

Hello,

I recently started to use this and I noticed that currently using batchWrite I cannot do conditions on my put requests.

Example:

const DocumentClient = new DynamoDB.DocumentClient({
  accessKeyId: 'hey',
  secretAccessKey: 'hey',
  endpoint: 'http://localhost:8000',
  region: 'us-west-2'
})
const table = new Table({
  name: 'testing123',

  partitionKey: 'PK',
  sortKey: 'SK',

  DocumentClient
})
const TestEntity = new Entity({
  name: 'Test',

  attributes: {
    PK: {
      partitionKey: true,
      type: 'string'
    },
    SK: {
      sortKey: true,
      type: 'string'
    }
  },

  table
})

console.log(
  JSON.stringify(
    table.batchWriteParams([
      TestEntity.putBatch(
        {
          PK: 'test',
          SK: 'test'
        },
        {
          conditions: [{ attr: 'PK', exists: false }]
        }
      )
    ])
  )
)

Logs:

{
   "RequestItems":{
      "testing123":[
         {
            "PutRequest":{
               "Item":{
                  "_ct":"2020-06-22T08:37:45.201Z",
                  "_md":"2020-06-22T08:37:45.201Z",
                  "entity":"Test",
                  "PK":"test",
                  "SK":"test"
               }
            }
         }
      ]
   }
}

Notice how there is no expression attached to this. I really need to have this feature as it is important I have it!

TypeScript support

Would love to see TypeScript types

Reopening this in reference to #14 since v0.2 is out.

aws-sdk dependency

aws-sdk is already included in Lambda NodeJS runtimes. If this lib is included, say, as part of a Lambda Layer, then ~50MB is added to the payload. Perhaps it can be added as a peer dependency instead?

Model or Facet

Would a better name for the core classes be a Facet instead of a Model ?

Model is very ORM'y , while Facet is more inline with NoSQL Workbench patterns.

Also, what's the status on the v0.2 branch (it looks like a great improvement).

eq check doesn't support false/empty strings

making an issue in case you want to tie it to #40 :)

Create "replicate attribute" option to schema definitions

So one of the interesting things I picked up in Rick Houlihan's talk at re:Invent (more details here), was the duplication of attributes in the SAME ITEM in order to make greater use of sparse indexes. Very, very, very cool stuff.

You could certainly use the default mechanism to replicate data attributes, but this seems like it could be a handy shortcut.

Proposed implementation

// Define schema
schema: {
  pk: { type: 'string', alias: 'id' },
  sk: { type: 'string', hidden: true },
  email: { type: 'string' },
  someDupeField: { $ref: 'email' }
}

The $ref (or something similar) would map the email value to the someDupeField attribute, always overwriting it whenever the data was updated in the main index.

v0.2 Documentation Typos

Typos are marked with bold:

1. Tables
   Tables represent one-to-one mappings to your DynamoDB tables. Then contain information about your table's name, primary keys, indexes, and more. They are also used organize and coordinate operations between entities. Tables support a number of methods that allow you to interact with your entities including performing queries, scans, batch gets and batch writes.

Throw a JS error instead of a literal

Looks like this is in 0.1 and 0.2: https://github.com/jeremydaly/dynamodb-toolbox/blob/master/index.js#L531

Instead of throw err, ought to be throw new Error(x) so normal try/catch stacks can inspect it.

I'll shoot some PRs to fix in a bit if you're cool with it :)

ExpressionAttributeValues is incorrectly present when conditions are used

Writing something like:

  const params = MyEntity.putParams(network, {
    conditions: [
      { attr: "PK", exists: false },
      { attr: "SK", exists: false }
    ]
  });

Results in an ExpressionAttributeValues: {}, which AWS rejects.

Workaround is to add:
delete params.ExpressionAttributeValues;

If you know they're empty.

Support AWS SDK for JavaScript v3

it will be nice to switch to the modularized AWS SDK for JavaScript v3 (https://github.com/aws/aws-sdk-js-v3) since it reached gamma status. AWS Amplify is already using it.

dynamodb-toolbox doesn't require the full AWS SDK and will reduce the size of the bundle.

FWIW, it's a bit premature since AWS SDK for JavaScript v3 doesn't support DocumentClient yet.

Entities need to list all attributes

If an entity does not list an attribute that exists in the database, get fails.

To make await entityName.get(item) work, in the definition of the entity entityName, it needs to list all attributes it may pull from the database for that particular item. Otherwise, it shows an alias of undefined error.

This is counterproductive for a single-table NoSQL design as often times, there is data that is there in the database, may get pulled by DynamoDB get, but the function doesn't need to be explicit about it.

Access pattern management

So this is a big one that would likely require query support (see #5). However, defining methods like getUserById, getOrdersByDate, or setUser seems to be a very common practice. These methods could be defined as higher level constructs that mapped to the underlying models. I would guess that a vast majority of get access patterns are just mapped to a query with some conditions and filters, and sets are a simple put operation (maybe with a transaction).

Need to think about this more, but it could also potentially (at least for the gets) help you keep track of your indexes and what they’re used for. Thoughts and ideas are welcome.

Auto-generated property "entity" interferes with update

A few properties are auto-generated when saving data, e.g., created, modified, and entity. However, while created and modified do not interfere with any subsequent update() operation, the entity property causes the update() to fail with the message "Error: Field 'entity' does not have a mapping or alias".

A workaround is to explicitly delete myObj.entity at some point in the chain.

Example Code:

const insertAndUpdate = async () => {
  const foo = { id: 'abc', name: 'Alfred' };
  await Foos.put(foo);
  const { Item: fooFromDB } = await Foos.get({ id: foo.id });
  fooFromDB.name = 'Bob'
  await Foos.update(fooFromDB); // FAILS - "Error: Field 'entity' does not have a mapping or alias"
  };

const FoosTable = new Table({
  name: 'test-Foos',
  partitionKey: 'pk',
  sortKey: 'sk',
  DocumentClient,
});

const Foos = new Entity({
  name: 'Foo',
  table: FoosTable,
  timestamps: true,
  attributes: {
    pk: { hidden: true, partitionKey: true, default: (data) => (`FOO#${data.id}`) },
    sk: { hidden: true, sortKey: true, default: (data) => (`FOO#${data.id}`) },
    id: { required: 'always' },
    name: 'string',
  },
});

CLI tool

I've been doing a lot of background work on this library and I've found myself writing some scripts to deal with different conversions and tests. I'm thinking a CLI component might be handy to generate models, compile code, import models from NoSQL Workbench (or export them), etc.

Way down on the priority list, but adding it here so I don't forget.

Entity.get() returns a parsed object under the key Item

Unless I'm doing something wrong, Entity.get() returns something like

{
    "Item": {
        "uuid": "8f19ef46-c9c2-4c68-ad19-bd22e82a0eaf",
        ...
    }
}

The keys under Item have been correctly parsed but this is not what I expected after having read the doc.

Should we change this behavior or the add it to the doc?

Map type entity attrs and schemas.

/Question
Interested to know what you will be doing with the TODO regarding object schemas.

For example I am using data and pdata entity level map type attributes. The pdata will be an object that I want projected into the indexes and the data is for pretty much everything else.

Say I want to update the pdata object type with a single attribute and not overwrite the existing record then it would be nice to be able to do a:

const userDbObj = { pk: sub, sk: 'User', pdata: { $set: { 'address.town': 'timbuktoo' } } }
await User.update(userDbObj)

if the 'address' attr has on the pdata object has not already been created then an error is thrown.

Thanks for the lib.

Issues with batch operations

First, thanks for the library, it makes it much easier to get started with DynamoDB.

There seem to be some problems with batch operations:

Batch get (https://github.com/jeremydaly/dynamodb-toolbox/blob/master/classes/Table.js#L786)

• parseBatchGetResponse call in the next function is called with a nextResult parameter which is a Promise, as it is not awaited on. This should probably be resolved before passing it to parseBatchGetResponse.

Batch write (https://github.com/jeremydaly/dynamodb-toolbox/blob/master/classes/Table.js#L959)

• Same issue as in batch get, nextResult should be awaited on.
• According to DynamoDB documentation the batchWrite operation returns UnprocessedItems instead of UnprocessedKeys. (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#batchWrite-property)

Question: How to query a table by partition key only

I'm implementing the northwind example in node using your library.

I'm trying to write the first query which is get employee by employee id (pk = employeeID) with the following code:

EmployeeModel.get({
	pk: 'employeeID#2'
})

but I get the following error:

'sk' is required`

Looking at the following test, it appears that querying by partition key only is not supported.

dynamodb-toolbox/__tests__/get.unit.test.js

Lines 45 to 47 in 92e3726

	it('fails when missing the sortKey', () => {
	expect(() => TestModel.get({ pk: 'test-pk' })).toThrow(`'sk' or 'type' is required`)
	})

Is this correct? Or am I holding it wrong?

v0.2 feedback - alias vs. map , and function to map data out on return

Great job on the toolbox. I appreciate it.

It was a little unclear from the documentation what the difference between alias and map is. I figured it out by looking into the source, but a little clarification would probably help others.

I was trying to use the map property as a way to translate a string in the DB table to an object output. org:{id:1234} into orgId:1234 in the db table and then back to org:{id:1234} when read. On the input using the default function solves it, but there seems to be no way to translate the output back when read. Is that correct?

Add delimiter option for composite sortKey generation

Currently composite sortKey generation uses the # symbol as a delimiter. This should be configurable.

default mapping doesn't support falsy values

Eg, 0 or "" or false aren't set if just provided like

{ type: "whatever", default: <falsy-value> }

It looks like some of the tests are accidentally expecting this behavior, but I'll look more later.

Workaround for now is to just use a function:

{ type: "whatever", default: () => <falsy-value> }

Global Secondary Indexes

Loving using the library so far Jeremy.

Can’t quite work out how to specify an IndexName so I can query a GSI, what have I missed?

Multi-model parsing

The parsing utility is handy, but right now you need to pass returned items to the correct model to parse them. I anticipated this originally, so I added the model option and defaulted it to true, which adds a __model attribute that stores the model name. But since there is no query support (see #5), the library doesn’t currently have a way to automatically map to the correct model.

This also needs some more thinking. A model registry would make this easier, but that would require changing the instantiation method to create a new instance of the DynamoDB Toolbox, and then use methods to build models. You could still expose just the Model class, but that might be substandard to something like:

const Toolbox = require(‘dynamodb-toolbox’)
const DDB = new Toolbox({ ... some global configs ... })

const MyModel = new DDB.Model(‘MyModel’,{ .. config... })

Models could them be automatically registered.

Then when you get Items back from DynamoDB, you could use a DDB.parse() that could use the registered models to automatically parse each item correctly. Thoughts and ideas are welcome.

Add support for querying by partition key alone

Requesting support for querying by just partition key: #11. This is a valid use case that would be leveraged when you split up a given entity over multiple items in a given partition to alleviate concerns about the 400kb item size limit.

e.g. You have a class record, and are concerned about the combination of students/teacher/TAs/assignments/tests/labs relationship data exceeding the limit of 400kb for a single item.

You can split all this data out in to multiple items with the same partition key and different sort keys. Then querying by just the partition key you can get all information regarding a given class while alleviating any concern about exceeding the 400kb item size limit.

Typescript Support

Do you intend to add Typescript support at some point in the future. Currently looking at this as an alternative to the awslabs/datamapper package as this supports some features that it does not currently and the only major thing I would be loosing (as far as I can tell), would be the strict typing.

In otherwords, do you have an ETA for the PR out for adding it as of yet?

autoExecute not recognized as valid option in put()

Expected Behavior: I can pass in an option to put() for autoExecute to override its behavior from the Entity.

Actual Behavior: I receive an error "Invalid put options: autoExecute".

Workaround: Turn off autoExecute at the Entity property level before the put() call, and set it back just afterward (see the example, below).

Sample code:

const updateNoAutoExecuteError = async () => {
  const foo = { id: '123', name: 'Carla' };
  const putParams = await Foos.put(foo, { autoExecute: false }); // ERROR - "Error: Invalid put options: autoExecute"
}

const updateNoAutoExecuteWorkaround = async () => {
  const foo = { id: '123', name: 'Carla' };
  Foos.autoExecute = false;
  const putParams = await Foos.put(foo);
  Foos.autoExecute = true;
  console.log(`putParams: ${JSON.stringify(putParams)}`); //  putParams: {"TableName":"test-foos", ...,  "name":"Carla"}}
}

const FoosTable = new Table({
  name: 'test-foos',
  partitionKey: 'pk',
  sortKey: 'sk',
  DocumentClient,
});

const Foos = new Entity({
  name: 'Foo',
  table: FoosTable,
  timestamps: true,
  attributes: {
    pk: { hidden: true, partitionKey: true, default: (data) => (`FOO#${data.id}`) },
    sk: { hidden: true, sortKey: true, default: (data) => (`FOO#${data.id}`) },
    id: { required: 'always' },
    name: 'string',
  },
});

Unexpected result with customize defaults of an attribute with a function

This is just a note of something I found to be unexpected. For (good or bad) reasons I want the sk to be the same as the pk, but I cannot use the name of pk in the input.

Example

A simple table

const Drawer = new Table({
  name: "Drawer",
  partitionKey: "pk",
  sortKey: "sk
})

and then we define an entity

Does not work

const Pants = new Entity({
  name: 'Pants',
  attributes: {
    id: { partitionKey: true},
    sk: {hidden: true, sortKey: true, default: (data) => data.id};
  },
  table: Drawer
});

await Account.put({id: 5});

This is because data does not contain id, but only pk:

data = {
  sk: [Function: default],
  _ct: [Function: default],
  _md: [Function: default],
  _et: 'Pants',
  pk: '5',

Works

const Pants = new Entity({
  name: 'Pants',
  attributes: {
    id: { partitionKey: true},
    sk: {hidden: true, sortKey: true, default: (data) => data.pk};
  },
  table: Drawer
});

await Account.put({id: 5});

Expectation

I expect data to contain the information as inputted i.e. in terms of the Entity and not the table 🤷‍♂️

Not sure whether more people would expect this, but I thought to submit an issue.

Projection Management / Mappings

Another issue that has crept up with large tables using lots of entities/facets, is the 20 attribute projection limit per GSI. I've used mappings in the past to use generic attribute names, thought I know this can be difficult to manage. Perhaps if the library had a way to indicate which GSIs each field needed to be projected to, then the mapping could be optimized. This might need to be a CLI feature, but I could see it being useful.

The other possibility is to add an "attribute id" of some sort to the entity/facet definition and then use that to determine which attributes to map data to. I think this would probably allow for quite a bit of future flexibility, but I'd need to test that out more.

This is another placeholder for now.

Query Support Proposal

Opening this in it's own ticket since it's sort of long and I wanted to allow it to be closed separately in the case I am completely insane.

I've been tinkering with this library for the last couple of days and trying to imagine how queries could work. A few things strike me as having potential here.

Model Introspection
First, the array definition for composite keys potentially gives the model some important information.For example, if only 1 of 2 parts of the composite exist, we may be able to infer that the query needs to have a begins_with() on the SK.

Potentially the GSI in use could be inferred by which data items are populated as well, though there may be a need to provide an explicit hint in some cases.

GSI Formatting
All GSI pk and sk schema formatting could probably follow the same schema formatting rules as the main pk and sk do. They could be dynamically constructed just like the main table SK constructed. The model would just need to flag those schema elements as belonging to a GSI. This could be done on the schema element itself, or in the main model definition.

As a side note, I think the gsi and table definition data could be lifted out of the model and defined separately for easier single table design.

DAT 403 Example
I'm a huge fan of Rick Houlihan's ReInvent presentations, as I know @jeremydaly is as well. I just spent a moment trying to envision how the employee model from his 2019 presentation might be constructed and queried. Here is what I came up with. There are likely errors

const Employee = new Model('Employee', {

  // Specify table name
  table: 'singletable',

  // Define partition and sort keys
  partitionKey: 'pk',
  sortKey: 'sk',

  // GSI here - type: gsi | local, defaults to gsi
  indexes: [
    { name: 'gsi1', partitionKey: 'gsi1pk', sortKey: 'gsi1sk' },
    { name: 'gsi2', partitionKey: 'gsi2pk', sortKey: 'gsi2sk' },
    { name: 'gsi3', partitionKey: 'gsi3pk', sortKey: 'gsi3sk' }
  ],

  // Define schema
  schema: {
    // primary keys
    pk: { type: 'string', default: (data) => `${data.employeeid}` },                    // Access Pattern 9
    sk: { type: 'string', hidden: true },
    sk0: ['sk', 0, { type: 'string', default: (data) => `${data.type}` }],              // Access Pattern 9
    sk1: ['sk', 1, { type: 'string', default: (data) => `${data.managerid}` }], 

    // GSI1
    gsi1pk: { type: 'string', default: (data) => `${data.employeeemail}` },             // Access Pattern 10
    gsi1sk: { type: 'string', default: (data) => `${data.sk}` },                        // Access Pattern 10

    // GSI2
    gsi2pk: { type: 'string', default: (data) => `${data.manageremail}` },              // Access pattern 15
    gsi2sk: { type: 'string', default: (data) => `${data.sk}` },                        // Access pattern 15

    // GSI3
    gsi3pk: { type: 'string', default: (data) => `${data.city}` },                      // Access pattern 14
    gsi3sk: { type: 'string', hidden: true },                                           // Access pattern 14
    gsi3sk0: ['gsi3sk', 0, { type: 'string', default: (data) => `${data.building}` }],  // Access Pattern 14
    gsi3sk1: ['gsi3sk', 1, { type: 'string', default: (data) => `${data.floor}` }],     // Access Pattern 14
    gsi3sk2: ['gsi3sk', 2, { type: 'string', default: (data) => `${data.aisle}` }],     // Access Pattern 14
    gsi3sk3: ['gsi3sk', 3, { type: 'string', default: (data) => `${data.desk}` }],      // Access Pattern 14

    // data
    type: { type: 'string', default: 'E' },
    employeeid: { type: 'string' },
    employeeemail: { type: 'string' },
    hiredate: { type: 'string' },
    managerid: { type: 'string' },
    manageremail: { type: 'string' },
    city: { type: 'string' },
    building: { type: 'string' },
    floor: { type: 'string' },
    aisle: { type: 'string' },
    desk: { type: 'string' }
    // ... other data attributes

  }
})

Then you could query the access patterns thusly:

Access Pattern 9 - Employee By ID
Since type maps to the first part of the composite sk, we can infer begins_with(). The main table's pk and sk are populated but none of the GSI's are, so we can also infer that this is a query against the table and not a GSI.

let item = {
  employeeid: 'employee22',
  type: 'E' 
}
let params = Employee.query(item)

Access Pattern 10 -Employee by Email
This is similar to the above example but the main table's pk will not be populated. Instead, GSI1's pk and sk will be populated, but no others, so we can infer that this query should be against GSI1.

let item = {
  employeeemail: '[email protected]',
  type: 'E'
}
let params = Employee.query(item)

Access Pattern 14 - employees by city, building, floor, aisle, desk
This one populates parts of GSI3, along with a partial gsi3sk, so we can infer that this query will be against gsi3, and will require a begins_with() on gsi3sk.

let item = {
  city: 'Atlanta',
  building: 'Davis West',
  floor: '4' 
}
let params = Employee.query(item)

Access Pattern 15 - Employees by Manager
This one is a little fuzzy because I am not entirely certain which elements in Rick's table contained the managerid - maybe it's the second element in the table's sk? I'm just guessing. But if that's true, this query will grab the employees that this person manages, again inferring gsi2 and begins_with() on gsi1sk.

let item = {
  manageremail: '[email protected]',
  type: 'E' 
}
let params = Employee.query(item)

There it is, warts and all. Hopefully at least some of this will add to the conversation.