endpointservices/mps3

Infraless Database over any s3 storage API.

mps3's Issues

Conformance test reliability

The minio test do not run reliably

R2 test

Shoudl work with R2 but not tested yet

`get` against a subscribed key should be served from cache

A subscribe keeps a key hot, so we don't need to poll for that case.

BUG: LOOKBACK is screwed up since timestamp change

AWS conformance test is looking back too much. Seems like it is accessing very old files and not auto-cleaning, probably because the manifest key format changed.

documentation

API docs plz

Bug: clock skew can break causal consistency

a participant can write into the future, making a local write mask the write until committed, when it then pops back into existence as the commited local write ends up temporally behind the existing future record. The invariant that local writes will end up at the head of the log is broken.

Refactor manifest representation

Its a bit overfitted to the KV of JSON in s3 use-case, we should allow content-types, query ranges and arbitrary web-urls under a common logical keyspace.

offline-first

We should serialize state for fast boot and for working offline

getAll method

GET should include a batch get which is a consistent snapshot. Would help testing too (e.g. parrallel puts)

Fix causal testing

We occasionally fail causaility. whats the issue?

minimizeLists = false, should disable the poll during syncing

for localfirst we don't need a poll file and the optionality is easier explained

Switch XML Parser to regex based parsing

None of the XML parsers are consistent across JS runtimes. We could use regex instead as the problem is fairly bounded

https://github.com/endpointservices/mps3/blob/main/docs/S3%20XML%20Escaping%20Cases.md

This would bring us closer to supporting more runtimes (like CF workers)

package bloat: 294kb due to aws-client dependancy

switch to aws4fetch

Allow request config to be mixed in asyncronously (e.g. Authorization: Bearer XXX)

For some proxies it would be useful to mix in cookies, and auth headers

useChecksum is incorrect

sha256 checksum is calculated incorrectly, workaround is "useChecksum: false" in options

If-not-changed

To build towards merge functionality, we should be able to conditionally write a document if there have been no changes to some other keys

Cache requests

We fetch the same immutable resources. Lets put an LRU in there

import / forget and move keys

To help people connect existing content into the database, we should support importing a key (just a manifest update). The inverse is forgetting a key, and to help archiving keys we should have a move.

list (keys) method

We have no way of enumerating what is in the DB

Manifest Garbage collection

The manifest leaves a trail, it should be automatically tidied up

Refactor to allow content and per Manifest s3 config

Content and manifest could be offline first, there should be no need to configure an s3 client. Also the credentials for each manifest may be different.

Bug: two clients with the same label interact badly over pending write queue

auto-clean references if the underlying content was deleted

If content is deleted behind mps3's back, the key should disappear from the manifest when detected.

Reverse timestamp so it sorts backwards for list-object-v2

We currently have to find a place to look forward from, it be more sympathetic if we do timestamps in reverse order

Poll if-none-match optimization is currently disabled

We tunred htis off during refactoring but its quite useful

gzip

we should optimize our bandwidth costs

auth

its not super clear how to secure this. We should provide examples

Fix causal consistency issues

Randomized testing is showing failures of the causal consistency checks. There seems to be a few race conditions when fetching content during notify.

BUG: List object returning empty for localfirst on demo page

Maybe prefix escaping?

BUG: If two local updates are applied to the same key, only the 1st notified a subscriber

On a cold start, client should read only as far back as indicated by the poll file

Currently the client seems to read the whole log on startup, but this is a bug.

Detect clock skew

We can use the AWS created at timestamp to validate the manifest timestamp on read.

Document sync algorithm

Now the algorithm has stabilised we should write it down

Structured tracing

We should log in a structured way, maybe something like zipkin with a trace_id passed through on reads which can be added on puts for end-to-end tracing across clients.

ListV2 fails when grabbing from unbooted DB

Surfaced using s3 as the upstream, might be just that, throws KeyNotFound which list object cannot parse.

Bug: 2nd sequential local write does not trigger notification

We have some bad logic for avoiding double firing local writes which ends up masking true negatively inhibiting things that should fire. (its got a test case "Read own write twice")

Make offline/online togglable

We should be able to switch to offline mode dynamically. This should terminate any hanging GET requests and throw Offline exceptions.

Bug: localfirst goes into a tailspin in browser

If you are using a proxy you don't need to set credentials

currently the config enforces credentials

list-objects should include trailing "/"