tywalch/electrodb

A DynamoDB library to ease the use of modeling complex hierarchical relationships and implementing a Single Table Design while keeping your query code readable.

ElectroDB

ElectroDB is a DynamoDB library to ease the use of having multiple entities and complex hierarchical relationships in a single DynamoDB table.

Please submit issues/feedback or reach out on Twitter @tinkertamper.

---

New: Documentation now found at ElectroDB.dev

ElectroDB's new website for Documentation is now live at www.ElectroDB.dev.

---

Introducing: The NEW ElectroDB Playground

Try out and share ElectroDB Models, Services, and Single Table Design at electrodb.fun

---

Features

• Single-Table Entity Isolation - Entities created with ElectroDB will not conflict with other entities when using a single DynamoDB table.
• Attribute Schema Enforcement - Define a schema for your entities with enforced attribute validation, defaults, types, aliases, and more.
• Easily Compose Hierarchical Access Patterns - Plan and design hierarchical keys for your indexes to multiply your possible access patterns.
• Simplified Sort Key Condition Querying - Write efficient sort key queries by easily building compose keys.
• Simplified Filter Composition - Easily create complex readable filters for DynamoDB queries without worrying about the implementation of ExpressionAttributeNames, ExpressionAttributeValues, and FilterExpressions.
• Simplified Condition Composition - Use the same interface to casily create complex readable mutation conditions for DynamoDB queries without worrying about the implementation of ExpressionAttributeNames, ExpressionAttributeValues, and ConditionExpressions.
• Simplified Update Expression Composition - Easily compose type safe update operations without having to format tedious ExpressionAttributeNames, ExpressionAttributeValues, and UpdateExpressions.
• Easily Query Across Entities - Define "collections" to create powerful/idiomatic queries that return multiple entities in a single request.
• Automatic Index Selection - Use .find() or .match() methods to dynamically and efficiently query based on defined sort key structures.
• Simplified Pagination API - ElectroDB generates url safe cursors for pagination, allows for fine grain automated pagination, and supports async iteration.
• Strong TypeScript Inference - Strong TypeScript support for both Entities and Services now in Beta.
• Query Directly via the Terminal - Execute queries against your Entities, Services, Models directly from the command line.
• Stand Up Rest Server for Entities - Stand up a REST Server to interact with your Entities, Services, Models for easier prototyping.
• Use with your existing tables - ElectroDB simplifies building DocumentClient parameters, so you can use it with existing tables/data.

---

Turn this

tasks
  .patch({
    team: "core",
    task: "45-662",
    project: "backend",
  })
  .set({ status: "open" })
  .add({ points: 5 })
  .append({
    comments: [
      {
        user: "janet",
        body: "This seems half-baked.",
      },
    ],
  })
  .where(({ status }, { eq }) => eq(status, "in-progress"))
  .go();

Into This

{
  "UpdateExpression": "SET #status = :status_u0, #points = #points + :points_u0, #comments = list_append(#comments, :comments_u0), #updatedAt = :updatedAt_u0, #gsi1sk = :gsi1sk_u0",
  "ExpressionAttributeNames": {
    "#status": "status",
    "#points": "points",
    "#comments": "comments",
    "#updatedAt": "updatedAt",
    "#gsi1sk": "gsi1sk"
  },
  "ExpressionAttributeValues": {
    ":status0": "in-progress",
    ":status_u0": "open",
    ":points_u0": 5,
    ":comments_u0": [
      {
        "user": "janet",
        "body": "This seems half-baked."
      }
    ],
    ":updatedAt_u0": 1630977029015,
    ":gsi1sk_u0": "$assignments#tasks_1#status_open"
  },
  "TableName": "your_table_name",
  "Key": {
    "pk": "$taskapp#team_core",
    "sk": "$tasks_1#project_backend#task_45-662"
  },
  "ConditionExpression": "attribute_exists(pk) AND attribute_exists(sk) AND #status = :status0"
}

electrodb's Issues

The where clause improperly handles the state of nested properties

Describe the bug
The where clause improperly handles the state of nested properties

ElectroDB Version
Specify the version of ElectroDB you are using
(e.g. 2.2.4)

ElectroDB Playground Link
https://electrodb.fun/?#code/PQKgBAsg9gJgpgGzARwK5wE4Es4GcA0YuccYGeqCALgUQBYCG5YA7llXWAGbZwB2MXGBDAAUKKwBbAA5QMVMAG8wAUT5V2AT0IBlTADcsAY1IBfbhiiSwAIkRwjVSzABGNgNzijUPrgVUGFwRSAF5bTShUDAB9AKC4aL4GSTgPcVAwABUGXABrITUNKk1hMW9ffxz8sDC+OBZVdS0AClEwJTb2sElYRAAuDq6u-iLNAZsAvNwbfE6h-UxcLB9xgEYZua7iDEMTccnchmlpG03TWaGGKicsF1QqPAHFTfaH5KeXruLpOHG-bD4AHMNkMhuQ0FhyDABk50J9zp8Dh9QV9ND8-jcgSCUWQ4BCoTCMHCUQiUdJLAArBxUZE476-Wz-LBYi448GoSFwaFgWFweGsoZwAAeyWkwVpKPpAwA5JIjtKBaDyVAfvIcLgJTieFZNTieWiGdK+KhJC5MArPkNSXqeVBdXSDTLjabzYqSW6utarWdFcz4ELHoMlZTqRqg2TcvahlwcAhuTZpLlsXrvDIoEsHgMANoTODJGwAXUtYC9oLyUa6McQ8byyZxwGAYCM5CupFTsgzpFycE0Qi4cjA0iYGgYSFwcgU3ZKaEw6uL7Xb6fYDJzyqpjhmtgOheLpnhZ061uUcWCJdEAEpPF4fH4eXnrGEbJJNLF72lyre4LIjJwwgARVsADo+CgFhmkvcQDlwQCZwwTROkAtdQ2aRQ3kkUxzwQlg6EwOBmhQsBhVFcUlAsKxCCoKASxLQhlGoOBCEBB5CBAqgVCFLA-CEDCagAPnDXEqCiPgwAAAwAEkUejmm1SRCC-KAf3PcwAEEADk-zAZpJKYvDKPk786GUsAAHkACUwEk1j2M4mhmko5Tz1Ew9MPaQDASgcCrygmD0DghCkMcXAULQjCsJw8h8KuJxaLAejGOYsBrI4riS3PPiBPIISMBEiSpIeZpoowQCiJkYJANkgzFKM1SNK0nSCqKkqRTKuBAP0wjDOM8zLMUZLbOCprSrFNqHIw0TPHaMK3I8rygA

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

const tasks = new Entity(
  {
    model: {
      entity: "tasks",
      version: "1",
      service: "taskapp"
    },
    attributes: {
      team: {
        type: "string",
        required: true
      },
      task: {
        type: "string",
        required: true
      },
      project: {
        type: "string",
        required: true
      },
      example: {
        type: 'map',
        properties: {
          from: {
            type: 'number',
          },
          to: {
            type: 'number',
          },
        },
      }
    },
    indexes: {
      projects: {
        pk: {
          field: "pk",
          composite: ["team"]
        },
        sk: {
          field: "sk",
          // create composite keys for partial sort key queries
          composite: ["project", "task"]
        }
      }
    }
  }
);

Expected behavior

Received:
"#example.#from <= :from0 AND (#example.#from.#to >= :to0 OR attribute_not_exists(#example.#from.#to))"

Expected:
"#example.#from <= :from0 AND (#example.#to >= :to0 OR attribute_not_exists(#example.#to))"

Question - How is the single table defined?

Looking at the various examples, I'm curious how you table is defined.

Is this correct?

{
  TableName: 'single_table_name',
  KeySchema: [
    { AttributeName: 'pk', KeyType: 'HASH' },
    { AttributeName: 'sk', KeyType: 'RANGE' },
  ],
  AttributeDefinitions: [
    { AttributeName: 'pk', AttributeType: 'S' },
    { AttributeName: 'sk', AttributeType: 'S' },
    { AttributeName: 'gsi1pk', AttributeType: 'S' },
    { AttributeName: 'gsi1sk', AttributeType: 'S' },
    { AttributeName: 'gsi2pk', AttributeType: 'S' },
    { AttributeName: 'gsi2sk', AttributeType: 'S' },
  ],
  GlobalSecondaryIndexes: [
    {
      IndexName: 'gsi1pk-gsi1sk-index',
      KeySchema: [
        { AttributeName: 'gsi1pk', KeyType: 'HASH' },
        { AttributeName: 'gsi1sk', KeyType: 'RANGE' },
      ],
      Projection: {
        ProjectionType: 'ALL',
      },
    },
    {
      IndexName: 'gsi2pk-gsi2sk-index',
      KeySchema: [
        { AttributeName: 'gsi2pk', KeyType: 'HASH' },
        { AttributeName: 'gsi2sk', KeyType: 'RANGE' },
      ],
      Projection: {
        ProjectionType: 'ALL',
      },
    },
  ],
  BillingMode: 'PAY_PER_REQUEST',
}

Service logger doesn't work

The following logger setup doesn't work

const service = new Service({ task, user }, {
   client,
   table,
   logger: console.log
});

The only way to get looger working is

.go({ logger: console.log })

or pass to each entity config

new Entity(UserModel, { logger: console.log });

Allow return values on patch/update request

Currently, whenever doing a patch or an update the response is null even when I set the params to have ReturnValues set to "ALL_NEW". I would like for the library to handle return values and parse the items for me.

Upserts?

Greetings BDFL of Electrodb!

Curious if Electrodb supports "upserts" out-of-the-box (create if not exists, otherwise patch/update).

We're looking to create an item if it doesn't exist with all of the same attributes (including those used in the PK and SK) that would be created with a put or create ... otherwise to just update the item (viz., all attributes except for those used in the PK or SK).

This can be done with a try { create } catch { patch }, but was thinking there might be something analogous to dynamodb's UpdateItem command.

Here's a gist showing what we've tried thus far.

Batched results TS typings are flipped.

The BatchGet two-dimensional array with the results of the query and any unprocessed records appear to be flipped; the actual data returned is as documented ([results, unprocessed]) however the typings show the opposite ([unprocessed, results]). So, to retrieve the correct data I have to cast the unprocessed element to the < ResponseItem >[].

e.g. of how I am getting around this:

        const batchAccountGroupIds = accountGroupIds.map(id => ({ accountId, accountGroupId: id }));
        const [accountGroups, _] = await accountGroupDao.get(batchAccountGroupIds).go(); 
        // ^ 'accountGroups' is typed as AllTableIndexCompositeAttributes<...> but returns ResponseItem data.
        // ^ '_' is typed as ResponseItem<...> 
        return accountGroups as AccountGroupEntity[];

(unsure if this applies to all the other batch operations)

[Feature Request] Required container attributes can have empty values

Modelling empty lists and sets by omitting the property is understandable from a storage perspective, but adds overhead to the business logic consuming the objects. As a modelling layer over DynamoDB, ElectroDB could abstract this storage requirement away.

Feature Request – The required property of list and set attributes should allow empty sets and lists, marshalling to and from null behind the scenes.

Required CustomAttribute not respected by TypeScript

Describe the bug
When using a required customAttribute it is still marked optional as far as TypeScript is concerned. However at runtime this will result in an Error.

ElectroDB Version
2.0.0

ElectroDB Playground Link
[(if possible) Use the ElectroDB Playground to recreate your issue and supply the link here to help with troubleshooting.](https://electrodb.fun/?#code/JYWwDg9gTgLgBAbzgUQHY2DAngGjgYygFMBDGIgYQFcBnGCEAQRhimACMry4BfOAMygM4AIiIAbIvlYQAJuxEBuAFDL8EVHTgwS7SXAC8orBCpQA+jr1FzqEiCJLV2METiNZIYKgBKEfUYIynDaWK4AXHAA5CSe3lHBcDRSxDCRdGyoAObKPM5hbgCqyVC+-m6BiS5EkTFxqAkh7CQ0wPjprN45ecrVcH4B7vUDbgA+cMVEpSOq6prwOjQA1jSGcKhEAO4o6JhYABSJQSEhIHISkccnIUS72JEiiysiOInXAG5TrRoPAIwvbxOJXebRqoieJDAYBEgJ4r2uZE6nHINEugJCwFkl1CEVEGS6Il48OucCEkkihFI5GodAYzCRXCIAB4RgA+fZXEmkogARyowGIWO0UCoRHRvAAlLDiRjULIiAAPIioxDisBsEAkKBYNFckJgJa6vUhfjACRCkQGgHGkLqcAQVrkSIAbREmJEAF1xSE4d6kobVTaBGbxBbltabXbII6wc6vca8lzfddE0SjtpdPo8hKVMoAPR5uAAFQKNEIwDACwAFt4Vtoa6tgKsIEs4MiUFAhFA4ABaHus3otFYAOkpZCIHMSmNqICwPcxCR4EuHWQg+ylyiAA)

Entity/Service Definitions

type AdminRole = {
  type: 'admin'
  secret: string
}

type UsernRole = {
  type: 'admin'
  basic: string
}

type Role = AdminRole | UsernRole

const tasks = new Entity(
  {
    model: {
      entity: "tasks",
      version: "1",
      service: "taskapp"
    },
    attributes: {
      id: { type: "string" },
      role: createCustomAttribute<Role>({
        required: true
      })
    },
    indexes: {
      primary: {
        pk: {
          field: "pk",
          composite: ["id"]
        },
        sk: {
          field: "sk",
          composite: []
        }
      },
    }
  },
  { table }
);

Expected behavior
Typescript should require role to be set to create/put

Errors

Additional context

Update Expressions with add/sub & map/list types

Hello, I see this in the readme:

Append/Add/Subtract/Remove updates capabilities
Complex attributes (list, map, set)

Out of curiosity do you have any ETA for these? Would be very useful to have, I tried to go through and see if I could add them myself but it was a bit hard to follow the code. If you are curious number 1 priority IMO is add/subtract because the any type works fine for now.

Thanks

Can I disable identifiers?

I want to query an existing dynamodb table, so I don't want to use edb_e, edb_v, and so on in the query.
Records in my table contain entity names at the pk field only, like entitiy#{entity id}.

Playground

{
    "TableName": "your_table_name",
    "ExpressionAttributeNames": {
        "#publishedAt": "publishedAt",
        "#pk": "pk",
        "#__edb_e__": "__edb_e__",  // <-- I don't want to use them because my records doesn't contain such attr.
        "#__edb_v__": "__edb_v__" // <--
    },
    "ExpressionAttributeValues": {
        ":publishedAt0": "2020-01-01",
        ":pk": "pose#",
        ":__edb_e__": "poses", // <--
        ":__edb_v__": "1" // <--
    },
    "FilterExpression": "begins_with(#pk, :pk) AND #__edb_e__ = :__edb_e__ AND #__edb_v__ = :__edb_v__ AND #publishedAt > :publishedAt0"
}

where clauses loses query typing

In the following example:

  const result = await thing
    .update({
      id: input.id,
    })
    .data((attr, op) => {
    })
    .where((attr, op) => op.eq(attr.code, input.code))
    .go({
      response: "all_new",
    })

There is a type error on response because it loses the fact that it's an update query. Removing the where clause fixes

Ability to make composite attribute from map item

I would like to be able to make an index that takes fields from a map item. For example say I have an item that belongs to a user, maybe a comment or something like that. I store the item in the comment entity as a partial of the user entity so I store it as a map, but trying to make an index of user.id does not work currently.

map fields are always initialized to `{}` even when not required and not defined

Describe the bug
Given an entity like this:

new Entity(
  {
    model: {
      entity: "foo",
      version: "1",
      service: "service"
    },
    attributes: {
      tenantName: {
        type: "string",
        required: true
      },
      configuration: {
        type: "map",
        required: false,
        properties: {
          alpha: {
            type: "any",
          },
          beta: {
            type: "any"
          }
        }
      },
    },
    indexes: {
      byTenantName: {
        pk: {
          field: "pk",
          composite: ["tenantName"]
        },
      },
    }
  },
  { table }
);

and initialization code like this:

    const item = await this.entity
      .create({
        tenantName,
      })
      .go();

I had expected only the tenantName field to be initialized.
However, configuration is initialized to an empty map:

{
  "pk": {
    "S": "$foo#tenantName_ttt"
  },
  "configuration": {
    "M": {}
  },
  "tenantName": {
    "S": "ttt"
  },
  "__edb_e__": {
    "S": "foo"
  },
  "__edb_v__": {
    "S": "1"
  }
}

I was surprised by this behaviour, and it broke a design that assumed the presence or otherwise of this field had meaning (something I can probably work around).

ElectroDB Version
1.11.1

ElectroDB Playground Link
Playground Link

Entity/Service Definitions
See above.

Expected behavior
I expected the field (e.g. configuration) to be absent until it was initialized.

Errors
n/a

Additional context
Speculation:

Haven't tried to seriously debug, but at first glance the _makeSet method for the MapAttribute in schema.js looks suspect in the way it creates an empty data object which is always returned.

_makeSet(set, properties) {
	this._checkGetSet(set, "set");
	const setter = set || ((attr) => attr);
	return (values, siblings) => {
		const data = {};
		if (values === undefined) {
			return setter(data, siblings);
		}
		for (const name of Object.keys(properties.attributes)) {
                  // ...
		}
		return setter(data, siblings);
	}
}

Missing License

The library is missing a license that permits use of it in other products.

The reason I am asking for this is that I would like to use this in a project and probably contribute to it (I am looking into additional update operations), but reading through the GH docs I found this section

However, without a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work.

Thanks for all the work you put into this library :)

Support/Query: Complex/Unnamed types makes Typescript use tricky

New to electrodb, so apologies if this is obvious or just fundamentally can't be improved.

I'm finding that the types created by electrodb are quite hard to work with in practice in standard Typescript with strict mode enabled and all the recommended eslint rules turned on:

• type signatures are complex and unpredictable
• named types really needed for function signatures, but hard to get

Simplest possible example

I have a class implementing a service interface, with the DynamoDB client passed into the constructor. I want to create an Entity (or a service) and store it as a member variable for use in methods. What type signature do I use for the member variable?

It's an Entity<something, something, something, something>.

I've found two acceptable approaches:

1: Observe and accept

Notice that while the type params are not obvious, the schema type is at least recognizable and the whole signature is manageable.

Create a type alias for the schema as just declare the entity using the IDE-provided signature:

   private readonly integrations: Entity<string, string, string, SomeSchema>;

2: The factory function gambit

Notice that Typescript will infer function return types and also provides ReturnType<...>

Make a factory function and exploit that feature:

export function SomeEntity(client: electrodb.DocumentClient, table: string) {
  return new electrodb.Entity(
    {
      // ... schema
    },
    {
      client,
      table,
    },
  );
}

  private readonly entity: ReturnType<typeof SomeEntity>;

  constructor(client: ddb.DynamoDBClient, table: string) {
    this.entity = SomeEntity(client, table);
  }

Commentary

This is obviously a trivial example but basically illustrates the issue I'm having:

electrodb generates complex type signatures with lots of parameters which presents difficulties when you want to reference them by name, as in this member variable.

Paging Example

So for the next example, let's say we want to expose a simple paging system, which might look like:

export interface ListSomethingOptions {
  nextPageToken?: string;
}

export interface SomethingStore {
  listSomethingAsync(tenantName: string, options: ListSomethingOptions): Promise<SomethingList>;
  // ...
}

The obvious thing for me to do is implement this using the page method, perhaps by encoding the returned page object into the nextPageToken (let's say as base64-encoded JSON for this example):

    const [next, items] = await this.entity.query
      .byIds({ tenantName })
      .page(decodePageToken(options.nextPageToken));

I need to implement decodePageToken, but what type does it return?
It's not obvious. The easiest workaround is to start turning off ESLint:

// eslint-disable-next-line @typescript-eslint/no-explicit-any
function decodePageToken(token: string | undefined): any {
  return token ? JSON.parse(Buffer.from(token, "base64").toString()) : undefined;
}

But this is probably better though it's equivalent:

function decodePageToken<T>(token: string | undefined): T | undefined {
  return token ? (JSON.parse(Buffer.from(token, "base64").toString()) as T) : undefined;
}

Item Mapping Example

Continuing on from the previous example, let's say we now want to map the stored items onto the SomethingList to be returned. Happily the type signatures mostly match, so writing a reusable map function should be easy... right?

Not so much - because again I've no type name for the fields I'm mapping from.

Doing an inline map is dead easy:

    const [next, items] = await this.entity.query
      .byIds({ tenantName })
      .page(decodePageToken(options.nextPageToken));

    return {
      nextPageToken: encodePageToken(next),
      items: items.map(function (item) {
        return {
          tenantName: item.tenantName,
          id: item.fooId,
          // ... about 10 more
        };
      }),
    };

Making it reusable looks much harder though.

Question

I think this is mostly happening because instead of starting with a user-constructed type and then applying magic - as with most popular ORMs, or the standard DocumentClient - we start with a schema and get generated types back.

Do you have some recommendations on the best way to tackle this?

Would it be possible to add more practical Typescript examples into the docs illustrating good usage patterns that minimise this sort of pain - I feel like there's a decent possibility I'm just "holding it wrong" :-)

Many thanks for your efforts on electrodb.

Allow sparse indexes

Is there a way to allow sparse indexes?

If an attribute of a composition key from a secondary index is missing we should still allow inserting/updating an entry this would just create the item without the item being also present in that index

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/bp-indexes-general-sparse-indexes.html

Missing composite attributes error when using patch + delete

Describe the bug

When using the patch method in combination with the delete operation, ElectroDB throws the "Incomplete or invalid key composite attributes supplied" error, despite all index attributes being provided, and the same parameters working correctly when passed to the update method instead.

// this works
tasks
  .update({ task, project, team })
  .remove(["description"])
  .params();

// this does not work
tasks
  .patch({ task, project, team })
  .remove(["description"])
  .params();

ElectroDB Version

2.3.2

ElectroDB Playground Link

https://codesandbox.io/p/sandbox/amazing-snowflake-3kt6bz?file=%2Findex.js

Entity/Service Definitions

I have added the ElectroDB playground model to the sandbox link above.

Expected behavior

It's expected that the patch method works the same as the update method, but with the implicit condition added to ensure that the document already exists.

Errors

ElectroError: Incomplete or invalid key composite attributes supplied. Missing properties: "team" - For more detail on this error reference: https://github.com/tywalch/electrodb#incomplete-composite-attributes
    at Entity._expectFacets (/project/sandbox/node_modules/electrodb/src/entity.js:2080:10)
    at Object.action (/project/sandbox/node_modules/electrodb/src/clauses.js:175:20)
    at Object.current.<computed> [as remove] (/project/sandbox/node_modules/electrodb/src/clauses.js:733:37)
    at file:///project/sandbox/index.js:141:4
    at ModuleJob.run (node:internal/modules/esm/module_job:193:25)
    at async Promise.all (index 0)
    at async ESMLoader.import (node:internal/modules/esm/loader:533:24)
    at async loadESM (node:internal/process/esm_loader:91:5)
    at async handleMainPromise (node:internal/modules/run_main:65:12) {
  _message: 'Incomplete or invalid key composite attributes supplied. Missing properties: "team"',
  ref: {
    code: 2002,
    section: 'incomplete-composite-attributes',
    name: 'IncompleteCompositeAttributes',
    sym: Symbol(error-code)
  },
  code: 2002,
  date: 1669637349858,
  isElectroError: true
}

Additional context

No additional context.

debugging/logging

Is there any way to enable a debug mode or some kind of logging so I can see all the expressions sent to DynamoDB?

Need a simple and accessible documentation site

Hello 👋🏼! This is going to be more of a feature request. I tried electrodb for a pet project and really loved this library to achieve 'Single Table Design'. Thank you so much!

However, as a new comer, though I could get started with electrodb without much hassle, I had trouble navigating the long README page especially during reference lookup during implementation. A simple documentation site (eg - Docusaurus, Vuepress, Mkdocs etc.) hosted in GitHub Pages would help onboard lot of new & experienced developers fairly quickly and easily.

A quick high-level structure would look like

|- Docs
   |- Introduction - Project Goals
   |- Features
   |- Installation
   |- Key Concepts
   |- Typescript Support
   ...
|- Tutorials
   |- A basic tutorial (entities, attributes, queries, filters)
   |- Advanced tutorial (collections, services, joins)
|- Community
   |- GitHub Discussions
   |- Issues
   |- Contribution Guidelines

The README has lot of content in a single file. Having this broken down into multiple pages - accessible via a sidebar would greatly help! :)

Error from validation failures.

if the getValidate(value) in src/schema has !valid - can you throw the validationError? I'm losing the error properties im throwing.

	isValid(value) {
		try {
			let [isTyped, typeError] = this._isType(value);
			let [isValid, validationError] = this.validate(value);
			let reason = [typeError, validationError].filter(Boolean).join(", ");
			return [isTyped && isValid, reason];
		} catch (err) {
			return [false, err.message];  <------ need the whole error
		}
	}

Incorrect validation using partition key as sort key in GSI

Describe the bug
I have an error I'm getting which I don't quite understand. I have the following dynamoDB with a GSI and a primary index. I try to represent this in electrodb with the following setup:

indexes: {
        asset: {
            pk: {
                field: "assetId",
                composite: ["assetId"]
            }
        },
        listingToAssets: {
            index: "listingToAssets",
            pk: {
                field: "listingId",
                composite: ["listingId"]
            },
            sk: {
                field: "assetId",
                composite: ["assetId"]
            }
        }
    }

I get the following error though:
Runtime.UnhandledPromiseRejection: ElectroError: The Sort Key (sk) on Access Pattern 'listingToAssets' references the field 'listingId' which is already referenced by the Access Pattern(s) 'asset' as a Partition Key. Fields mapped to Partition Keys cannot be also mapped to Sort Keys. - For more detail on this error reference: https://github.com/tywalch/electrodb#inconsistent-index-definition

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

{
    model: {
        entity: "assetId",
        version: "1",
        service: "task"
    },
    attributes: {
        assetId: {
            type: "string",
            default: () => uuid(),
        },
        listingId: {
            type: "string",
            required: true
        },
        status: {
            type: "string",
            required: true
        },
        type: {
            type: "string",
            required: true,
        },
        createdAt:{
            type: "number",
            required: true
        },
        updatedAt: {
            type: "number",
            required: true
        },
        ttl: {
            type: "number",
            required: true
        }
    },
    indexes: {
        asset: {
            pk: {
                field: "assetId",
                composite: ["assetId"]
            }
        },
        listingToAssets: {
            index: "listingToAssets",
            pk: {
                field: "listingId",
                composite: ["listingId"]
            },
            sk: {
                field: "assetId",
                composite: ["assetId"]
            }
        }
    }

Expected behavior
No validation error. GSI shouldn't have validations for what's in the primary index.

Errors

Runtime.UnhandledPromiseRejection: ElectroError: The Sort Key (sk) on Access Pattern 'listingToAssets' references the field 'listingId' which is already referenced by the Access Pattern(s) 'asset' as a Partition Key. Fields mapped to Partition Keys cannot be also mapped to Sort Keys. - For more detail on this error reference: https://github.com/tywalch/electrodb#inconsistent-index-definition 

Additional context
Discord discussion link:
https://discord.com/channels/983865673656705025/994984119224246303/1041444793349329078

How to handle electrodb errores gracefully inside a lambda function.

Hi, i have lambda function like this

export async function deleteSubcription(event: any) {
  const sub = paramValidation.safeParse(event.body.params);
  if (!sub.success) {
    throw createError(422, "Invalid input parameters!");
  } 
  try {
    let response = await SubscriptionEntity.delete({
      subId: sub.data.subId,
      planId: sub.data.planId,
    })
      .where((attr, op) => `${op.lte(attr.ref_count, 0)}`)
      .go();
    return {
      statusCode: "200",
      body: JSON.stringify({
        message: response,
        error: "",
      }),
    };
  } catch (error: any) {
    return {
      statusCode: error.statusCode,
      body: JSON.stringify({
        message: "",
        error: error,
      }),
    };
  }
}
export const handler = commonMiddy(deleteSubcription);

Electrodb throws an error but i am unable to catch it in my function code?
Any think i have to configure to catch errors ?

Attribute validation not invoked when using data update method

Describe the bug
When updating attributes via the data() update method, attribute validations are not invoked.

ElectroDB Version
2.1.2

Entity/Service Definitions

Using this entity, note that there is a validate method on the prop1 attribute, which will fail if the attribute does not contain the substring foo.

{
    model: {
        entity: "my_entity",
        service: "my_service",
        version: "my_version"
    },
    attributes: {
        prop1: {
            type: "string",
            validate: (value) => !value.includes("foo"),
        },
        prop2: {
            type: "string"
        }
    },
    indexes: {
        record: {
            pk: {
                field: "pk",
                composite: ["prop1"],
            },
            sk: {
                field: "sk",
                composite: ["prop2"],
            },
        }
    } 
}

If we do this, then the validation fails as expected, and we get an error.

MyEntity.update({ entityID }).set({ prop1: "bar" });

However, if we do this instead, then the validation does not fail. In fact, the validate callback is not invoked at all.

MyEntity.update({ entityID }).data((attributes, operations) => {
  operations.set(attributes.prop1, "bar");
});

Expected behavior
Attribute validations should be invoked when using the update methods inside the data method.

Additional context
I understand that the underlying logic of the data method is probably different from the standard set, update etc, but the data method is really helpful for performing conditional operations. For example, I might call the set or remove operation based on the value of an attribute. This behavior is not possible using the default method chaining technique.

Issue with link to Playground

Describe the bug
The link does not work on line 1468 in the readme.

Expected behavior
The link works

Errors

404

Additional context
I'd fix this, but I'm not exactly sure what model you want to show this feature. Sorry :(

Ability to parse items (DynamoDB Streams)

Looking to start using dynamodb streams would be nice to be able to parse the items it sends me using electrodb. Could be something on the entity level and service level. For example maybe I only really care about one item type for a specific operation that occurred so I do entity.parse(dynamoItem) and it could return the parsed item if it is of that entity type or null/error if not. Then on the service level, it could return type: 'entityName' item: parsedItem

Default values in nested attributes

Currently, for nested "map" attributes, the attribute property default is only evaluated when its parent is provided by the user via a put or create method. In its current implementation, a user can defined an empty object as a default value on their map attribute, however that will not cause a the object to then be populated with the default values of the nested attributes on that "map". This can be counter intuitive when from a user experience when the map attribute will always exist for a given entity, and and the only way to apply defaults would be to supply an empty object.

A better approach would be evaluate default values outward, starting with root maps and extending to nested attributes. If a default value (or callback) is not supplied or returns undefined, then further nested defaults will not be applied.

An example for this implementation can be found here.

Modeling help request

Hey Tyler.
I looked high and low for this in the README and in /examples. It seems like it should be straightforward.

I wanted to model the following:

Creating the top half of the model was easy. I then applied what I know about creating comments on a post or tickets on a project to addProductToWarehouse but I was immediately stumped. Can you offer some assistance when you have some time? It's not obvious to me how this should work.

https://electrodb.fun/?#code/PQKgBAsg9gJgpgGzARwK5wE4Es4GcA0YuccYGeqCALgUQBYCG5YA7llXWAGbZwB2MXGBDAAUKKwBbAA5QMVMAG8wAUT5V2AT0IBlTADcsAY1IBfbhiiSwAIkRwjVSzABGNgNzijUPrgVUGFwRSAF5bTShUDAB9AKC4aL4GSTgPcTgAD1l5MG9fBQAFZ1RHNQ0qTTAwvjgWVXUtAAowUTAlVrawSVhEAC52zs79TFwsH36bAEYbfA7B-nLNCaLYEqoZuc7iDEMTCbl4DFwNwdNZwYYqJywXVCo8fsVNtulixwBJGEfnzorpOAmfmwfAA5idBp1yGgsOQvmAnOhzhCzj8kilvhDfpp-oDrqDwZioagYXA4Qi4EjTpTOvBcEZsNINOMBpj4diAbYgVh8dS2ijMa9jBynqy2TjbHxUJIXJgCZ1+RDvKh1BjWX8OTZJdLZbywAq+dTufAMg8WYNgMB4XQsEIEDaaGBXqtHLgfm0LWB2AByIRSf4YLgOBS3BRQADWDE0PydMDWuFVArDCdZXBwCDhNmkYblmO8Migo3u-QA2gBdXUGt1EJNmlNpjO4bMVzp52SFjnFzNvKifGzlqumH76zosJhwOiRYgrWMu5ODbwIYKOMZ8Caj8gT1BT7vHZtGzITEGjabNrNziGpxAZo9YSZZnOKqxt9gdmzr8eTuC9-ui4eDRvnoMl7poex6Ng+85PgWL4ll2zo9jAfbNoOmLDihepIoocTBCiACU6RZHICh5H4YAAOpjpuxBlFoVRgDUdQ0RUzQdCKnTdPACDnsMRwrhMJ4-AsWgTBRG6fg+2y7BqBwjHKw6XNcIammxgzvlRX5wipELqriwJgs2RIkmSGCIkOuposKVY6ZyeL6VWhmwv05K6n+YAMDAMDkLg8a1oM1k2FyPL2XA0KOfCJkUmZPySAwSQgpggFihqgV2ah1LDvuJo+VpbRqZ+2VVh6HA2mAdp+EIeVbngLSih63q+jImCBo4YAhmA4aRlWC5Lkyq62JVxC7lWZ6+Re9YTPezZtK20FFmAZbIc2AGjUB42ck2VbTVB7awQNGlIQOUWYnt05xol3VBnx-WUZ+p0uhBnoCAetg3gJoojTlY1XqBt6TZtuTbTB81vjdVXfotVbLZ9q3fS9YEbaKW35jt80-qy6FUps6EKlhgQ4aI+EEdkCgMLgmh8EY3DKsuPi5OQlxwHdVCNBZ-QpYQtL0lgjIrgA-GztlgAAPvRlAIEigomPzovahgwui4uhBKuo0tajKcsi5Ki64bW5BUFEfBgEzTGaAAdPScAM40n0xmsnyAlYcAALSoMSiGEBalBYDAjS4eZySRZinMMr1-TB9zvXCyLNgPpLHJx1HYAAAy6srVD9GnidJ1juGmyCUC+6Ig5E0RblkxTVMU5HFsM6JH5Vdb9EB4Q7meXgQimI8TfokQtktx5Xm4NLKXy1r4uOt2nzD4Lmti3qOtsXrBvkaD1ENBUpsdDX9zW5sreD-0+-t4nMfUhZ1J7fbnKO07UAcJgLtuzMYAel7Pt+x0pi5-nhfFx6AAq1oKpyDDEIEE9MqAAEJRCZGJmXcmlMuDU0jq3CAsUGDxQwI0S+cJ2bdwBClBemwl4YENnXdSJtTaoGkDAK2n0cHpVzsQZmn0YpxQSvgxhecC7KC8rIXwGoGCLkSLUGw88i7iA9O8L01gkGLkqAgAsCgJx1CoFAMAMB1HFXjKIKRrBYr3BgPCdRFtLb3EIG1aR1gGCOFQEIhAlRqG0I0KCXRloyKMAUO8fR6hPRCFwI7YqoJSpYDDHAU2ES3FgC3NyEEVpSC2w+DAQgzD4lgH0EI9AUSoBcH0WJMGRi2AcDcgDZUVAYGERyKTBBlcaaG1bkzf+UByGfmwavDSAs9KEESQhTpsTCCs17npIhgwSGG2NuvM2Ti6HRknsknOptmG72Ou0z4uoPRp36AAai2Qsn++FTBAA

Get EntityType from Schema, not Entity

With my current code I can get the UserType with EntityItem<typeof users>. My question is how can I get UserType from userSchema? My use case is I'd like to put userSchema and UserType into a common package so I can share the UserType between backend and frontend code. Thanks!

import { Entity, EntityItem, Schema } from "electrodb";

const userSchema: Schema<string, string, string> =  {
    model: {
      entity: "user",
      version: "1",
      service: "main",
    },
    attributes: {
      userId: {
        type: "string",
        readOnly: true,
        required: true,
      },
      email: {
        type: "string",
        required: true,
      },
      firstName: {
        type: "string",
        required: true,
      },
      lastName: {
        type: "string",
        required: true,
      },
      createdDate: {
        type: "string",
        default: () => new Date().toISOString(),
        readOnly: true,
      },
      updatedDate: {
        type: "string",
        readOnly: true,
        set: () => new Date().toISOString(),
        watch: "*",
      },
    },
    indexes: {
      primary: {
        pk: {
          field: "pk",
          composite: [],
        },
        sk: {
          field: "sk",
          composite: [],
        },
      },
    },
  };

const users = new Entity(userSchema, entityConfig);

type UserType = EntityItem<typeof users>;

`.params()` doesn't work with TransactWriteItems and v3 SDK

Describe the bug
Trying to use a list of .params() with TransactWriteItemsCommand results in the cryptic error

TypeError: Cannot read properties of undefined (reading '0')

After some investigation, this seems to be the cause:
aws/aws-sdk-js-v3#3365 (comment)

.params() returns something like this:

{
    "Delete": {
        "Key": {
            "pk": "$sites#siteid_01gk976gqx9cqq63htxj0pbyer",
            "sk": "$devices#device_1#deviceid_01gqnz2pjv3cx8kx242dkjsw13"
        },
        "TableName": "dev-table-ng",
        "ConditionExpression": "attribute_exists(#pk) AND attribute_exists(#sk)",
        "ExpressionAttributeNames": {
            "#pk": "pk",
            "#sk": "sk"
        }
    }
}

But if I understand correctly it should be something like this:

{
    "Delete": {
        "Key": {
            "pk": {"S": "$sites#siteid_01gk976gqx9cqq63htxj0pbyer"},
            "sk": {"S": "$devices#device_1#deviceid_01gqnz2pjv3cx8kx242dkjsw13"}
        },
        "TableName": "dev-table-ng",
        "ConditionExpression": "attribute_exists(#pk) AND attribute_exists(#sk)",
        "ExpressionAttributeNames": {
            "#pk": "pk",
            "#sk": "sk"
        }
    }
}

ElectroDB Version
1.11.1

docs typo

Under Create Record the docs say:

In DynamoDB, put operations by default will overwrite a record if record being updated does not exist. In ElectroDB, the patch method will utilize the attribute_not_exists() parameter dynamically to ensure records are only "created" and not overwritten when inserting new records into the table.

Should "In ElectroDB, the patch method will utilize..." say "In ElectroDB, the create method will utilize..."?

Immutable collections

Before I explain the concept of "Immutable collections" let me just say thank you for writing this library. I investigated a couple of others and this one is the best I found. I only tried some bits and went through the docs but so far it is great.

I use the "Immutable collections" concept in a few of my projects and they work well so I was wondering if I can get your feedback and see if that would be possible to implement in electrodb. If not, then I will handle that by creating another service to do that.

The way you have implemented collections is impressive but I see one drawback which is, all collections use GSIs. GSIs are great but nothing is free. These are my concerns around GSIs

• to read data from collections all GSIs require to project all data from the table, which increases the cost of storing data, and additional cost for all transfers, every time we update an attribute on a table all GSIs will be updated and I believe DynamoDB will be coping the whole item to all GSIs,
• on top of that backups, global tables and point-in-time recovery add quite a bit of complexity and with a high number of GSIs this might be a problem
• in a scenario where you have 20 or more GSIs, in a very large table our costs might be high and some processes can be slower (backups, restoring data)

So my solution is to maintain my own index in the same table under different PK, SK for all attributes which are immutable and can be queried. Let's say we store organizations and each organization has a type (SMALL, MEDIUM, LARGE), for this example let's assume that attribute never changes. So after we create an organization we will never change that attribute, in that case, I don't want to use GSI for this but I still want to be able to query organizations based on type. To do that while creating an organization record I also create another item which will be that index (no other attributes, just PK, SK). Since that attribute is immutable I will never have to update that index (if that was GSI all updates on organization record would trigger data replication to all GSIs).

To query all organizations by type I would use that index. The only downside here is that I need to use two queries (assuming I don't fetch more than 100 records and they all are under 16MB). The first query would get all records for a specific organization type and the second query will query all organizations' details.

// Organization entity
{
    "pk": "org#1",
    "sk": "...",
    "type": "SMALL"
}
// OrganizationByType entity
{
    "pk": "orgByType#SMALL",
    "sk": "org#1"
}

So the main benefits are:

• decrease the number of required GSIs which should reflect costs. This solution is not recommended for attributes that can be changed.
• assuming the table is running on-demand and all our indexes have a unique PK, stored data is minimum we don't have to worry about provisioning (in case of GSIs we use GSI overloading so we don't know what kind of load will be used and what provisioning is required, setting up all GSIs on-demand will be expensive)
• this solution would work even for attributes that can be changed, but it would require two updates

Drawbacks:

• when we create an organization we also need to create a type record, so to keep consistency at 100% we need to use a transaction to insert both at the same time, which let us support up to 25 immutable attributes, more than that would require some kind of compensation process to be implemented
• when query we need to use at least two queries

Let me know what are your thoughts.

Model versioning use cases

ElectroDB supports model versioning, however, I would like to find out what are the use cases for that. When I change the versions, all new data will be saved with the new version but I am not able to fetch data using the previous one.

Could anyone provide their way of using model versioning?

empty composite key

Hey @tywalch, first of all, great to know that SST is using ElectroDB as their tool of choice.

I've been trying to use an empty composite primary key like this:

But it gives me this message: instance.indexes.primary.pk.composite does not meet minimum length of 1.

The reason being that, in some entities, I want them to be partitioned only by the service name (which is added to the pk automatically), nothing else. I could just have a field emptyKey or something, but I just thought whether there's a cleaner way of doing that - or even if you are interested in supporting that use case.

[Feature Request] attributes of 'number' type, enumerated 'as const'.

An example is easiest to explain:

{
  attributes: {
    ratingNumStars: {
      type: [0, 1, 2, 3, 4, 5] as const,
      default: 0
    }
  }
}

This would allow specifying the field as a number, with improved type safety (compared to using a validate method alone), while keeping numerical sorting.

Are there any plans to support transactions

Thanks for the library and the work you have done here 👍

I wanted to see if I can use this library for my use case but there are cases where I need to use transactions. Is there any way I can do that with the library?

`Cannot find module '@aws-sdk/client-dynamodb'` with v2 sdk

Describe the bug

use electrodb in a project uses dynamodb v2 sdk, it shows the error.

Error: Cannot find module '@aws-sdk/client-dynamodb'
Require stack:
- /Users/miyamonz/my-project/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/commands/BatchExecuteStatementCommand.js
- /Users/miyamonz/my-project/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/DynamoDBDocument.js
- /Users/miyamonz/my-project/node_modules/@aws-sdk/lib-dynamodb/dist-cjs/index.js
- /Users/miyamonz/my-project/node_modules/electrodb/src/client.js
- /Users/miyamonz/my-project/node_modules/electrodb/src/entity.js
- /Users/miyamonz/my-project/node_modules/electrodb/index.js

ElectroDB Version
2.3.3

Entity/Service Definitions

It seems it's not related to this issue, but just in case.

const handPose = new Entity(
  {
    model: {
      entity: "handPose",
      version: "1",
      service: "hand",
    },
    attributes: {
      poseId: {
        type: "string",
        required: true,
        readOnly: true,
      },
      createdAt: {
        type: "string",
        required: true,
        readOnly: true,
        default: () => new Date().toISOString(),
      },
      updatedAt: {
        type: "string",
        required: true,
        default: () => new Date().toISOString(),
      },
      data: {
        type: CustomAttributeType<HandPoseData>("any"),
        required: true,
      },
    },
    indexes: {
      poses: {
        pk: {
          field: "pk",
          composite: [],
          template: `hand#pose`,
        },
        sk: {
          field: "gsipk1",
          composite: ["poseId"],
          template: "poseId#${poseId}",
        },
      },
    },
  },
  { table: TABLE_NAME, client }
);

Expected behavior

it should work with v2 client without installing v3 client.

BTW, installing v3 though I don't use it in my project that uses v2, then the error disappeared, of course,
but importing v2 and v3 causes a type error, so It would be helpful if you could address this issue.

Additional context

electrodb/src/client.js

Line 1 in 95187d3

const lib = require('@aws-sdk/lib-dynamodb')

I think this line is the problem. it imports v3 client. it should be imported dynamically.

Old dependency on aws-sdk

Describe the bug
I have recently switched over to AWS SSO auth, with a sso-session section in ~/.aws/config. This results in the following error when I try to run tests:

ElectroError: Profile is configured with invalid SSO credentials. Required parameters "sso_account_id", "sso_region", "sso_role_name", "sso_start_url". Got region, default_region, sso_session, sso_account_id, sso_role_name
Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html - For more detail on this error reference: https://github.com/tywalch/electrodb#aws-error

The mentioned missing parameters are used in the "Legacy non-refreshable configuration", which is not recommended according to the reference link (https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html).

I assume this error occurs because support for the token-based configuration wasn't added until [email protected] (see https://github.com/aws/aws-sdk-js/blob/master/CHANGELOG.md#212530), but electroDB depends on version 2.630.

ElectroDB Version
2.1.1

ElectroDB Playground Link
N/A

Entity/Service Definitions
N/A

Expected behavior
AWS config with "SSO token provider configuration" (as recommended at https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html) should work without errors.

Errors

ElectroError: Profile is configured with invalid SSO credentials. Required parameters "sso_account_id", "sso_region", "sso_role_name", "sso_start_url". Got region, default_region, sso_session, sso_account_id, sso_role_name
Reference: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html - For more detail on this error reference: https://github.com/tywalch/electrodb#aws-error

keys in lowercase

hey guys. PK's and SK's content is being written in lowercase to dynamodb - can we have an option to disable that?

Here we have key templates like:
template: 'processId:${processId}:sourceProcessStepId:${sourceProcessStepId}',

Error in `parseModel` -> `normalize indexes`

We are getting this error message when trying to put an item in to our database, from some debugging and logs we can see that when looping through the access patterns that there are extra access patterns after our defined ones which are return undefined for the index and hence throwing the issue above, some cloudwatch logs below show the extra "access patterns"

Add support for AWS SDK v3 (`@aws-sdk/client-dynamodb`)

Would be great to have support for AWS SDK v3.

import { DynamoDBClient } from '@aws-sdk/client-dynamodb'

const client = new DynamoDBClient({})

Thanks!

Examples clarification

In the "library" example https://github.com/tywalch/electrodb/blob/master/examples/library/index.ts
we have the following queries with comments

// get all copies of books by the authors last name
author.query
    .writer({authorLastName: 'king'})
    .go();

// this same query can be used to get unique books
// (instead of individual copies) because this
// entity is denormalized
author.query
    .writer({authorLastName: 'king'})
    .go()

// get all books by last name and a
// partial first name.
author.query
    .writer({authorLastName: 'king'})
    .begins({authorFirstName: 's'})
    .go();

// get all books by the full name of the writer
author.query
    .writer({authorLastName: 'king', authorFirstName: 'stephen'})
    .go();

I tried that example, and none of them can return a list of books. They always return just the author data. Are those examples correct?

Remove an element from an attribute of type list

Describe the bug
Unable to remove a specific item from a list property/attribute using ElectroDb

ElectroDB Version
2.3.5

ElectroDB Playground Link
Link

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

const trackers = new Entity(
  {
    model: {
      entity: "tracker",
      version: "1",
      service: "trackerapp"
    },
    attributes: {
      gameId: {
        type: "string",
        required: true
      },
      subscribers: {
        type: 'list',
        required: true,
        items: {
          type: 'map',
          required: true,
          properties: {
            emailAddress: { type: 'string', required: true },
            thresholdPrice: { type: 'number', required: true },
          },
        },
      },
      createdAt: {
        type: "number",
        default: () => Date.now(),
        // cannot be modified after created
        readOnly: true
      },
      updatedAt: {
        type: "number",
        // watch for changes to any attribute
        watch: "*",
        // set current timestamp when updated
        set: () => Date.now(),
        readOnly: true
      }
    },
    indexes: {
      games: {
        pk: { field: "pk", composite: ["gameId"] },
        sk: { field: "sk", composite: [] }
      }
    }
  },
  { table }
);

Expected behavior
An ability to remove a specific element from tracker.subscribers list. DynamoDb's documentation on Removing Items from a List.

Errors
I've tried to do it with .remove method on .patch but that just takes a string/property name and complains that

Attribute "subscribers" is Required and cannot be removed

For more detail on this error reference https://electrodb.dev/en/reference/errors/#invalid-attribute

I've also tried the following

.data((attr, op) => `${op.remove(attr.subscribers)}[2]`)

but that throws the same error.

The other way I've tried is just manually entering the query in .data like so .data((attr, op) => 'REMOVE subscribers[2]') but the operation string gets dropped and doesn't end up executing.

Additional context

Local Secondary Indexes

Hey guys, great library! Any plans on supporting LSI's?

Feature Request: Simple example docs

Describe the bug
This is more a feature request than a bug.

ElectroDB Version

    "electrodb": "^1.11.1",

ElectroDB Playground Link
I'm not convinced this link is useful but here you have it.

Entity/Service Definitions
Again not really sure this is useful.

{
        model: {
          version: "1",
          entity: "Team",
          service: "tickets",
        },
        attributes: {
          teamId: {
            type: "string",
            required: true,
            readOnly: true,
          },
          name: {
            type: "string",
            required: true,
          },
        },
        indexes: {
          primary: {
            pk:{
              field: "pk",
              composite: []
            },
            sk: {
              field: "sk",
              composite: ["profile", "name"]
            }
          },
        }
      },
}

Expected behavior

I'd like to start with a thanks for your work on this. I have a ton of confidence that learning it and getting better at it will be worthwhile.

I expected to be able to consult the docs and find some simple examples that were straightforward to understand. I wanted to learn electrodb to keep along with SST's adoption. My goal was to build the following table

PK	SK	name	ticket
tickets	team_profile_1	team1
tickets	team_1_ticket_1		1
tickets	team_1_ticket_2		2
tickets	team_profile_2	team2
tickets	team_2_ticket_1		1

This gives me the following access patterns

getAllTeams -> pk: tickets sk: begins_with("team_profile")
getAllTicketsByTeam -> pk: tickets sk: beginsWith(team_${teamId}_ticket)

that seemed straightforward enough to practice. Anyway, I've been immediately stumped. I'm confident that I'll work it out. I just need to read more carefully and understand ElectroDb better. Work with the runkits etc...

Typically, I've used this as reference and sort of built up my own query by hand. https://dynobase.dev/dynamodb-nodejs/

I'm excited to pick up this tool and work through it. But I wanted to share that the docs might currently be a limiting factor in adoption. I'm happy to stick with it. There's a ton to get through right now but I wanted to provide you what someone who is just trying to pick this up. You've obviously done a ton of work, and I'd be happy to help in any way to make this more approachable for n00bs like myself.

Additional context

I'm not a savvy DynamoDB user and am here via SST's adoption of this tool

Filter by size of a list attribute

Describe the bug
Filter the query by using something equivalent of size(#attribute) method in DynamoDB.

ElectroDB Version
2.3.5

ElectroDB Playground Link
Link

Entity/Service Definitions

{
  model: {
    entity: "tracker",
    version: "1",
    service: "trackerapp"
  },
  attributes: {
    category: {
      type: "string",
      required: true
    },
    gameId: {
      type: "string",
      required: true
    },
    subscribers: {
      type: 'list',
      required: true,
      items: {
        type: 'map',
        required: true,
        properties: {
          emailAddress: { type: 'string', required: true },
          thresholdPrice: { type: 'number', required: true },
        },
      },
    },
    createdAt: {
      type: "number",
      default: () => Date.now(),
      // cannot be modified after created
      readOnly: true
    },
    updatedAt: {
      type: "number",
      // watch for changes to any attribute
      watch: "*",
      // set current timestamp when updated
      set: () => Date.now(),
      readOnly: true
    }
  },
  indexes: {
    category: {
      pk: { field: "pk", composite: ["category"] },
      sk: { field: "sk", composite: ['gameId'] }
    },
  }
}

Expected behavior
Be able to filter based on the size of subscribers attribute/property. Something like this:

trackers.query.category({category: 'adventure'})
  .where((item, op) => op.gt(op.size(item.subscribers), 0))
  .go();

Errors

op.size is not a function

Additional context
Add any other context about the problem here.

Model Version is required, even though documentation says it's optional

My error: Runtime.UnhandledPromiseRejection: ElectroError: instance.model.version is required
The documentation: (optional) The version number of the schema, used to namespace keys

I very desperately do NOT want to namespace on model version and do not want to leverage this feature. However, I can't use this library without doing so.

I would deeply appreciate either:

• a docs update and a feature to disable automatic injection of model versions into keys
• or make model versions truly optional

Thanks for making this library- I'm very excited to actually get to use it!

BatchGet returns zero results

Describe the bug
Performing a .get({table: "TABLE_NAME"}) on a Model, when passing the table name in as an option does not return any results:

const response = await Segment.get([
  { tenantId, siteId, segmentId: "1" },
  { tenantId, siteId, segmentId: "2" },
]).go({ table });

I have debugged the code and I can see that the DynamoDB Doc Client does return results - however Electro does not surface those results back to the application code.

When I run in debug mode I can see that the code in entity.js:571 is not being run because the table variable appears to have not been set:

		if (response.Responses[table] && Array.isArray(response.Responses[table])) {
			const responses = response.Responses[table];
			for (let i = 0; i < responses.length; i++) {
				const item = responses[i];
				const slot = orderMaintainer.getOrder(item);
				const formatted = this.formatResponse({Item: item}, index, config);
				if (slot !== -1) {
					resultsAll[slot] = formatted.data;
				} else {
					resultsAll.push(formatted.data);
				}
			}
		}

In this code above, the response.Responses contains a correctly formatted array of results. But, because table is undefined it is not able to process those results. Therefore the return is an empty array.

I have looked into the code but I cannot see where the table should have been set. I suspect it is because I am passing table as an option to the go method.

ElectroDB Version
2.2.0

ElectroDB Playground Link
N/A - this issue is with the handling of the DynamoDB Doc Client response - not the formatting of the initial request.

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

const Segment = new Entity({
  model: {
    entity: "Segment",
    version: "1",
    service: "TourContent",
  },
  attributes: {
    tenantId: {
      type: "string",
      required: true,
    },
    siteId: {
      type: "string",
      required: true,
    },
    segmentId: {
      type: "string",
      required: true,
      default: uuidv4,
      readOnly: true,
      validate: /^[0-9a-fA-F]{8}\b-[0-9a-fA-F]{4}\b-[0-9a-fA-F]{4}\b-[0-9a-fA-F]{4}\b-[0-9a-fA-F]{12}$/i,
    },
    name: {
      type: "string",
      required: true,
    },
    createdAt,
    updatedAt,
    state: {
      type: ["active", "inactive"] as const,
      required: true,
    },
    image: {
      type: "map",
      required: false,
      properties: {
        bucket: {
          type: "string",
          required: false,
        },
        key: {
          type: "string",
          required: false,
        },
      },
    },
  },
  indexes: {
    _: {
      pk: {
        field: PARTITION_KEY,
        composite: ["tenantId", "siteId", "segmentId"],
      },
      sk: {
        field: SORT_KEY,
        composite: [],
      },
    },
    site: {
      collection: "site",
      index: GSI1_INDEX_NAME,
      pk: {
        field: GSI1_PARTITION_KEY,
        composite: ["tenantId", "siteId"],
      },
      sk: {
        field: GSI1_SORT_KEY,
        composite: ["segmentId"],
      },
    },
  },
});

Expected behavior
The results which DynamoDB Doc Client responds with is processed successfully and the items are returned to the application code

Errors
N/A

"Property x is missing in type" Typescript error when using setters to set properties that are in composite keys

Describe the bug
Given this minimal example gist that reproduces this error, I would expect that I should be able to create new items and autogenerate the id at the entity level.

Instead, I get a Typescript error:

This makes sense -- the id is required in order to create an index and I do notCo encounter Typescript errors like this on attributes that are not part of the Composite Attributes.

But it seems to make sense to me that I should be able to generate the id using an Attribute Setter, since downstream logic shouldn't concern itself with that.

The workaround, of course, is to do something like the following:

 id: {
    type: "string",
    set: (id) => {
            if (!id) {
              return randomUUID();
            }
          },
          required: true,
          readOnly: true,
     }        

And then manually pass in an id when creating a new entity:

  const data = {
    name: "blah",
    id: randomUUID()
  };

  const item = await entity.create(data).go();

Or just not use an attribute setter for the id property. But that seems inelegant; I think ElectroDB Entities should be able to handle setting the id in a case like this, rather than relying on downstream logic. Since it's readOnly, I'm not worried about the warning in the readme about using attribute setters on Composite Attributes.

I'm not sure what the best solution, if any, is here, but happy to chip in if needed.

ElectroDB Version
2.3.3

ElectroDB Playground Link
Example

Entity/Service Definitions
Include your entity model (or a model that sufficiently recreates your issue) to help troubleshoot.

    {
      model: {
        entity: "device",
        version: "1",
        service: "DeviceService",
      },
      attributes: {
        id: {
          type: "string",
          set: () => randomUUID(),
          required: true,
          readOnly: true
        },
        name: {
          type: "string",
        },
      },
      indexes: {
        id: {
          pk: {
            field: "pk",
            composite: ["id"],
          },
        },
      },
    }

Expected behavior
I expect to be able to delegate the setting of the id attribute to the ElectroDB entity, even though it's part of my primary key.

Errors

error TS2345: Argument of type '{ name: string; }' is not assignable to parameter of type 'PutItem<string, string, string, { model: { entity: string; version: string; service: string; }; attributes: { id: { type: "string"; set: (id: string) => string; required: true; readOnly: true; }; name: { type: "string"; }; }; indexes: { ...; }; }>'.
  Property 'id' is missing in type '{ name: string; }' but required in type 'Pick<CreatedItem<string, string, string, { model: { entity: string; version: string; service: string; }; attributes: { id: { type: "string"; set: (id: string) => string; required: true; readOnly: true; }; name: { ...; }; }; indexes: { ...; }; }, { ...; }>, "id">'.

Feature Request: Helper Type to Extract Entity Types from Schema object

Given the following schema I'd like to have a helper types that's exported from electrodb that allows me to extract types so that I can use them in my frontend:

import { Putable, Getable, Schema, Updateable } from "electrodb";

const userSchema: Schema<string, string, string> =  {
    model: {
      entity: "user",
      version: "1",
      service: "main",
    },
    attributes: {
      userId: {
        type: "string",
        readOnly: true,
        required: true,
      },
      email: {
        type: "string",
        required: true,
      },
      firstName: {
        type: "string",
        required: true,
      },
      lastName: {
        type: "string",
        required: true,
      },
      createdDate: {
        type: "string",
        default: () => new Date().toISOString(),
        readOnly: true,
      },
      updatedDate: {
        type: "string",
        readOnly: true,
        set: () => new Date().toISOString(),
        watch: "*",
      },
    },
    indexes: {
      primary: {
        pk: {
          field: "pk",
          composite: [],
        },
        sk: {
          field: "sk",
          composite: [],
        },
      },
    },
  };

export type User = Getable<typeof userSchema>;
export type CreateUser = Putable<typeof userSchema>;
export type UpdateUser = Updateable<typeof userSchema>;

Requested types: Getable, Putable, Updateable.

Inspiration from kysely:

import type { ColumnType, Insertable, Selectable, Updateable } from "kysely";
import type { ManagementRoleName } from "./management-role.js";
import type { ProductRoleName } from "./product-role.js";
import type { NullableColumn } from "./helper-types.js";

export interface UserTable {
  accessExpirationDate: string;
  adOid: string; // PK UUID
  altEmail: NullableColumn<string>;
  createdDate: ColumnType<string, never, never>;
  department: string;
  edipi: string;
  email: string;
  firstName: string;
  iaCompletionDate: string;
  jobTitle: string;
  lastName: string;
  middleInitial: NullableColumn<string>;
  organization: string;
  phone: string;
  updatedDate: ColumnType<string, never, never>;
}

export type User = Selectable<UserTable>;
export type CreateUser = Insertable<UserTable>;
export type UpdateUser = Updateable<UserTable>;

Earlier issue: #165

Partition key without composite

I am trying to create a simple index to get a collection of all records (this will be used with pagination). I don't want to use scan as it is expensive with large tables.

My index looks like that:

all: {
      index: 'gsi2pk-gsi2sk-index',
      pk: {
        field: 'gsi2pk',
        template: '$users'
      },
      sk: {
        field: 'gsi2sk',
        composite: ['id'],
        template: '${id}'
      },
    },

then I just want to call

db.entities.users.query.all().go({ limit: 100 });

but it looks like that is not possible.

I use typescript so:

• composite is required on pk and it can't be an empty array
• if I have a composite key then I need to use it inside the template
• I tried to disable TS errors but then I am required to pass an object when I call .all()

db.entities.users.query.all({ anything: null }).go({ limit: 100 });

that works but requires that dummy object { anything: null } to be passed

the only way I got that working was to add a pseudo attribute

all: {
      type: 'string',
      hidden: true,
      required: false,
      default: '',
    }

and setup my index as

all: {
      index: 'gsi2pk-gsi2sk-index',
      pk: {
        field: 'gsi2pk',
        composite: ['all'],
        template: '$users${all}'
      },
      sk: {
        field: 'gsi2sk',
        composite: ['id'],
        template: '${id}'
      },
    }

and call

db.entities.users.query.all({ all: '' }).go({ limit: 100 });

that works as expected but there is too much boilerplate code and I still need to pass that object to .all() method

is there any other way to achieve that?