The vlcn-orm from vlcn-io

Fixup query layer generics

The client facing API always presents the right generics. Internal to the framework they aren't always correct, however.

The problem has to do with queries being able to return different projects.

E.g.,
id, node, edge, hop with nodes projections.

And maybe edges being partially optimized (rolled into prior expressions) as well.

more complex demo app

Todo lists barely require any interesting functionality.

Re-write strut to use Aphrodite and/or some other non-trivial apps.

type breakage in db resolvers?

see aphrodite-browser-starter index.ts where we had to add the any cast.

Fix type errors in generated code when using ts strict mode

gen in Base
and whereX in query methods

Problem on generics where passing T | null when T is expected.

tease apart schema responsibilities

see linked diff

add tests for each case uncovered in this commit

570574b

create codegen test framework

test that the codegen re-generates some expected output

Add connection pool support

wa-sqlite serializes reads that are on the same connection.

We should update the sqlite connectors to:

have a single write connection (all writes are always serialized even at the sqlite level)
have N read connections

To allow for parallel reads.

Add an example app that enables perf logging

#19
#20

https://github.com/tantaman/aphrodite/tree/main/packages/instrument

sign generated code to prevent manual modifications

Abstract and simplify edge definition?

Should we allow defining edges without defining id fields??

Foo as Node {
  blahId: ID<Blah>
} & OutboundEdges {
  blah: Edge<Foo.blahId>
}

to:

Foo as Node {
} & Edges {
  blah: OneToOne<Blah>
}

but this removes control from the developer and feels a bit opaque.

enable composing multiple build steps

Schema -> Models -> TSGraphQL -> ...

Implement cursor support

related to #24 which needs to be done first (before/after support).

validation - enforce correct index creation based on uniqueness constraints on edges

Measure write time performance

See linked project description.

allow manual sections in generated code

Implement take/before/after/orderBy hoisting

in SQLExpression.ts

Prioritize list of storage backends to support

Given the focus on local-first software the initial list would be:

IndexDB
SQLite

And to not completely preclude adding a server as a peer for server side syncing

Postgres
Maria

What else? The list should be fairly limited until we've proven the concept.

validation - check that edge and field names don't conflict

Use smithy as the specification language?

https://awslabs.github.io/smithy/index.html

in-memory only support (allow no backing db whatsoever)

Should be trivial given current use of chunk iterables.

`wantData` / `withNode` / `select`/ ...

Some sort of modifier to allow returning the node that was traversed when doing nested traversals.

Similar to:

https://entgo.io/docs/eager-load

Idea similarity to Riffle

This kind of reminded me of https://riffle.systems/essays/prelude/ . Maybe you find the ideas presented there interesting and relevant.

Feel free to close this issue for lack of actionable items.

i64 support in browser & node

@databases connections do not support 64 bit ints. They silently get truncated.

better-sqlite returns everything as bigints. Not great.

sqlite-3 has a fork that returns ints as strings if they'd overflow.

Single runtime package

Bundle all the various runtime-ts packages into one so its easier for clients to use. Would require updating codegen to import from this single package.

Right now we're using our own 64bit int id generation scheme (loosely based on https://dev.mysql.com/doc/refman/5.6/en/miscellaneous-functions.html#function_uuid-short). Problem is peer ids are not so easily assigned as server ids. Given the issue with assigning ids to peers (and clock skew between peers), I've done no formal verification that this won't collide.

Security would be another concern. What are the attack vectors in the p2p space when the id space is partially deterministic? Data that is shared between trusted peers wouldn't have an issue. "Public" apps where any peer can connect could have malicious peers that try resetting their clocks to clobber data. Authz rules and hmac'd replications should prevent this, however?

We could use nanoid but that has the concerns raised here https://www.percona.com/blog/2019/11/22/uuids-are-popular-but-bad-for-performance-lets-discuss/

maybe that doesn't matter given local first apps likely won't have datasets large enough for this to become an issue.

Given all the unknowns with sid and the probably small perf hit with nanoid -- we should go with nanoid for now.

remove files that should no longer be generated

If schema definitions are removed, remove the generated code too.

Enable more complex types in fields

Fields can be:

primitives
ids
arrays
maps

Lets add:

sets

Explore adding:

other model instances

Explore:

thrift, protobuff & json-serializable things

Validation step - Check that required inverses exist

Handle querying from the opposite side of a junction edge

Thoughts on GraphQL as the schema lang

Should we just extend the GraphQL schema spec and use that for Aphrodite schemas?

Thought about this off and on for a long time.

A prompt that got me thinking in this direction one more time was GraphJin:

https://graphjin.com/ | https://github.com/dosco/graphjin

read after optimistic write & optimistic reactivity

If a user strictly performs optimistic writes (without awaiting for the write to complete) they can issue a read that doesn't return what they expect.

The current design requires hitting the database to fulfill any query. So issuing a write that you don't await can result in a later query not returning the data in that write.

We can:

Update our query layer to understand if there are pending writes. If so, await them before returning the read.
^-- I think this might already be the case accidentally when using wa-sqlite-connector as all those operations are serialized

When we make changes for #46 we'll have to consider how it impacts this task.

Reactive queries do not update optimistically. This might not be a big deal but could lead to strange behavior where a component lags behind another. When tackling #49 we can address parts of this problem. E.g., by making single hop reactive queries resolve themselves in-memory.

Allow printing of query plans

So users can understand if all clauses are optimized or not.

Log a warning when a non-optimized query is run against a dataset of >1k rows.

aphrodite schema language server

like #57 but a language server component for auto-complete.

Schemas should be singletons

schema watch command

watch and auto-regenerate on schema change

Queries that reference other fields (whereExists, whereRaw)

I know it's early on, but I'm just tinkering with this. One thing I'd like to express in a query is a predicate which is related to another field on a model.

Example use case: a grocery list app which lets you change the quantity of items. Suppose you've already purchased 6 eggs, but you increase the desired quantity to 12. In order to express a query for "purchased items," I want to query all items where item.purchasedQuantity >= item.desiredQuantity.

The scope here would only be predicates relevant to other fields on the same model, not traversing edges or anything like that.

Potential route to have a single table in sqlite that contains all the atoms of the application.

The benefit is simpler migrations? maybe...
We can do the TAO model which alleviates most migration problems while also enabling nodes.

Smarter Reactive Queries

Currently we just re-run a query whenever one of its queried tables is updated.

This is good enough for now (based on current perf numbers and being in beta) but in the future if we need row-level reactivity some ideas are:

For single hop queries, we can run expressions in-memory against the modified/created/deleted node.

For multi-hop we of course lose the ids of the nodes we joined on. We have a few options:

👎 Require materialized views so all queries are single hop
1. Horrible devx
2. Horrible to maintain
3. We could automate this if all queries are statically known ala Relay.
👎 In the background, fetch ids of hops.
what if there are thousands or millions of these for a given hop?
👍 Model the query path as a graph.
1. We know:
  1. Start node
  2. End node(s)
  3. Some updated node inside the path (the target of the mutation)
2. We can use that to run simpler queries across subsections of the path to understand if the mutation target is part of the original query

Add union/intersect/concat/take/before/after/etc. to base query type

IndexedDB support?

While pure sqlite is great it is quite a bit of bloat to bring into the browser.

Is it worth supporting indexeddb directly?

Some reasons to do it:

faster first time load
less complex application setup (no sql files to generate, simpler table creation process, simpler db startup)

Not to:

Entirely new SourceExpression type to support
Entirely new query plan optimizer to write
CRDT layer would need to be re-implemented for IndexedDB

vlcn-io / vlcn-orm Goto Github PK

vlcn-orm's People

Contributors

Stargazers

Watchers

Forkers

vlcn-orm's Issues

Recommend Projects

Recommend Topics

Recommend Org