streamich/json-joy

JSON CRDT, JSON CRDT Patch, JSON Patch+, JSON Predicate, CBOR, MessagePack, UBJSON, JSON Reactive RPC, JSON-RPC 2.0, JSON Pointer, JSON Expression, JSON Type

json-joy's Issues

JSON URL

Implement JSON URL or similar format where objects and arrays are discriminated.

Reactive-RPC hardening

• Make sure on protocol error WebSocket is disconnected or can gracefully handle garbage data.

JSON-Rx improvements

• Do not send unsubscribe message from client when error is received.
• Add support for batching multiple messages into an array.
• Add automatic batcher, which waits for 1ms to combine multiple messages.

Compiled JSON Pointer

• Add ability to pre-compile JSON Pointer to JavaScript code.
• Compilation could also return the complexity of the JSON Pointer. Complexity could be measured in the number of JavaScript expressions.

Reactive-RPC optimization

• Allow methods to return serialized JSON or MessagePack output.
• Use best output type, depending on the serialization format that server uses.

DDS

Explore and implement JSON CRDT using Microsoft DDS (Distributed Data Type) approach.

JSON Patch match

• Enable JSON Patch match operation, but with safe RegExp implementation.
• Use Re2 library for regular expressions.

`json-type` JSON codegen

• Write codegen for efficient JSON serialization given schema.
• Add .isInteger or type: 'u8' | 'u16' | 'u32' ... annotations to "num" json-type.
• Add "safe string" annotation to json-type. Which does not need to be escaped in:
  • JSON.stringify
  • and when encoding UTF-8 string in MessagePack
• Maybe a string could also have a type, say type: 'ascii' | 'utf8' ....
• Add README.

Validation

Currently only json nominal encoding for JSON Patch+ operations is validated. Create a way to validate the operation patch regardless of the way it is serialized.

Incoming format of operation patch:

• json
• compact
• binary

References

• Add ability to identify any resource in the system using a string.

`json-type` to JSON Schema

• Implement function which converts json-type schema to JSON Schema.

Reactive-RPC performance testing

• Add benchmarks.
• Optimize static call hot path.

Timeout

Close streaming calls after some timeout due to inactivity

JSON Size

• Implement function which computes the size of JSON for a given plain JavaScript object.
• Use json-random for fuzz testing.
• Benchmark agains serializing to JSON string and taken the length of that string.

Reactive-RPC: request validation

• Add request validation
• For streaming requests, validate every request value
• Make validation optional

Reactive-RPC: Redundant stream complete message

When client unsubscribes from streaming Reactive-RCP response, the server should not send ResponseCompleteMessage as it is implicit.

json-filter

• Implement JSON filtering language: given a filter specification in JSON and a JSON document, the filter returns true or false.
• Return schema complexity estimate at compilation time? Complexity could be expressed as the number of JavaScript expressions.
• On schema evaluation, return the number of JavaScript expressions evaluated?
• Codegen for json-expression.
  • ExprGet
  • ExprEquals
  • ExprNotEquals
  • ExprIf
  • ExprAnd
  • ExprOr
  • ExprNot
  • ExprType
  • ExprBool
  • ExprNum
  • ExprInt
  • ExprStr
  • ExprStarts
  • ExprContains
  • ExprEnds
  • ExprDefined
  • ExprIn
  • ExprMatches
  • ExprCat
  • ExprSubstr
  • ExprLessThan
  • ExprLessThanOrEqual
  • ExprGreaterThan
  • ExprGreaterThanOrEqual
  • ExprBetweenNeNe
  • ExprBetweenNeEq
  • ExprBetweenEqNe
  • ExprBetweenEqEq
  • ExprMin
  • ExprMax
  • ExprPlus
  • ExprMinus
  • ExprAsterisk
  • ExprSlash
  • ExprMod
  • ExprRound
  • ExprCeil
  • ExprFloor
• Ability to provide default value for json-expression "get" operator if value not found.
• json-expression should throw on any undefined values? As undefined is not part of JSON and is unique to JavaScript.
• Implement benchmarks
  • codegen vs evaluator
  • vs JSON Logic
  • vs JSON patch
• Implement AWS SNS filtering rule compiler to json-expression
• Add README. Name it JSON Expression.

---

• See AWS SQS and AWS SNS filtering language for inspiration.
  • https://docs.aws.amazon.com/sns/latest/dg/sns-message-filtering.html
• Consider using JSON Predicate specification.
• Common Query Language (CQL)
  • https://portal.ogc.org/files/96288
• Consider JSON Logic
• Maybe JSON Schema and JSON Predicate are already good enough for filtering?
• It could be something like a List language in JSON.

---

It could be called json-expressions and implement basic expressions, like:

['and',
    ['starts', 'mu.bl/:posts-', ['get', '/req/id']],
    ['<', ['len', ['get', '/req/batch']], 5]
]
 
['add', '/tmp/userId', ['val', 'xxxxxxxx']]

['and',
    ['==',
    	'document-123',
     	['get', '/channel']],
    ['==',
     	'mousemove',
     	['get', '/data/type']],
    ['<',
     	123234343433,
     	['get', '/data/timestamp']],
    ['in', ['get', '/data/customer_interests'], [[
      "rugby",
      "football",
      "baseball"
   ]]]
]

Alternatively, a more advanced version of this could treat everything as function:

['map', [[1, 2, 3]], ['*', 2]] // [2, 4, 6]

Reactive-RPC: pre-call checks

Execute pre-call checks such as authentication and authorization, and throttling.

Clients

Implement JSON-Rx and Binary-Rx clients.

• JSON-Rx
  • Over WebSocket
    • JSON using text messages
    • MessagePack using binary messages
    • Use isomorphic WebSocket implementation
    • Allow periodic "ping" messages
    • Allow "auth" message
  • Over Server Sent Events
  • Over NDJSON
• Binary-Rx
  • Over WebSocket
    • Allow payload as blob with custom codec
    • Implement MessagePack message encoding
• JSON-RPC
  • Over WebSocket
  • Over HTTP POST request
  • Over HTTP GET request
  • Over Server Sent Events
  • Over NDJSON
• Multiplexing over SSE and NDJSON

Authentication

• Allow authentication over Authorization HTTP header.
• Allow authentication over access_token URL search param.
• Allow authentication over WebSocket JSON-Rx "token" notification message.
• Allow authentication over WebSocket sub-protocol header.
• Allow authentication over WebSocket Authorization header.

json-binary

Add binary data Uint8Array support to JSON.stringify and JSON.parse. Implement JSONB.stringify and JSONB.parse.

• Ability to encode a POJO with binary data into JSON POJO.
  • Convert binary blobs (Uint8Array, Buffer, etc..) to data URI. Binary data in JSON can be encoded as string, which starts with data:application/octet-stream;base64,.
  • Convert MessagePack extensions to data URI. MessagePack extensions can be encoded as a string, which starts with data:application/msgpack;base64;ext=<num>, or data:application/octet-stream;base64;ext=<num>,.
  • Support JsonPackValue class.
• Ability to decode JSON with Base64 data URLs back into binary?
  • Convert blob data URIs to Uint8Array.
  • Convert MessagePack extension data URIs to MsgPackExtension.
  • Support JsonPackValue class.
• Fail on invalid Base64 data.

Compiled JSON Patch

• Add ability to pre-compile JSON Patch to JavaScript code.

Challenges:

• It should be compatible the regular implementation.
  • Should throw the same errors.
  • Should return the same result.
  • Ideally, should support all the same commands.
• It should not increase the bundle size by much.

JSON-Rx with native JSON objects

Implement server part of JSON-Rx which uses JSON array objects instead of serialized strings.

`json-type` MessagePack codegen

• Write codegen for efficient MessagePack serialization given schema.
• Add README.

JSON Logic / JSON Magic

• Implement JSON Magic.
• Add JSON Magic as a predicate operation to JSON Patch.
• Introduce variable type in JSON Magic that extracts a value using JSON Pointer from the value on which JSON Patch operation is being performed.

OT

Implement operational transformations

• For string operations.
• Remove str prop from string operations, leave length.
• Add json0 type operation.
• Add json1 type operation.
  • With text-unicode subtype.
  • With text subtype.
• Add text type operation.
• Add text-unicode type operation.

Custom data

Allow storing custom payload on operations.

Reactive-RPC HTTP

Expose Reactive-RPC server using HTTP for uWebsocket.js server

• HTTP POST
• HTTP GET

json-pack partial parsing

• Add ability to parse only one level of the object and leave other nested data structures as JsonPackValue blobs.

Reactive-RPC: imperative API for RpcServer

Add programmatic API for RpcServer for static and streaming methods to execute server methods programatically.

`json-type` randomizer

• Implement function which can generate a random JSON given json-type schema.
• Use that randomizer for fuzz testing.

Set operations

Implement operations that treat arrays as sets. See json-patch/json-patch2#8 (comment)

MessagePack builder

MsgPack to JSON

Write a fast MessagePack to JSON converter, which does not do an intermediate step to load JSON into an in-memory representation, but directly writes to JSON.

json-equal

Implement various fast JSON equality functions.

• deep equal
• shallow equal
• code generated deep equal
• code generated shallow equal

Change license

Candidates:

• Elastic License v2
• MongoDB SSPL
• Elastic License 2.0
• BSL 1.1, like Replicache
• One of Creative Commons licenses

Base64

Make encoder and MsgPack encoder to be able to write directly into Base64 string.

This should be an optimization, instead of writing to a Buffer and then converting to Base64.

String op name in compact JSON Patch encoding

Allow in compact JSON Patch encoding to use string op names.

Instead of:

export type CompactAddOp = [opcode: OPCODE.add, path: string | Path, value: unknown];

allow:

export type CompactAddOp = [opcode: OPCODE.add | 'add', path: string | Path, value: unknown];

Add Shelf implementation

https://onp4.com/@vadim/p/fg2a7qjwrl

Amazon Ion

• Investigate Amazon Ion.
• See if any of that would be useful for json-joy.
  • Implement binary format codec

Reactive-RPC SSE

Expose Reactive-RPC server using HTTP streaming requests for uWebsocket.js server

• Server Sent Events
  • GET
  • POST

Add HAM implementation

https://gun.eco/docs/Conflict-Resolution-with-Guns

MsgPack validator

Write a quick validator, which checks that blob contents is a valid MessagePack object.

• It should not parse/decode the MessagePack blob, should just check that it is valid.
  • Should handle the case when multiple Message pack values follow each other. For example, when number follows an object {}5. It should allow the user to fail such cases.
• It should run faster than the MessagePack decoder.

Reactive-RPC binary

• Allow binary encoding of Reactive-RPC to support custom binary payloads or MessagePack payloads.

`json-type`

• Add enum type
• Add explicit or type
• Add ref type
• Support customValidators property as in JSON Schema "format", which allows to provide extra validator.
• Improve .Object() interface to receive fields as list.
• Improve t. builder to allow easy type ID specification.
• Binary encoding for JSON json-type-serializer.
• Switch function, which compiles an optimized version if used given number of times.
• Way to organize and compile many json-type entities.
• Recursive refs.
• json-type to JSON Schema.
• json-type to TypeScript.
  • To multiple types for readability.
• Add README.
• Ability to add types to RPC.
• Ability for an obj type to extend another obj type?
• Fast validation of or type, which holds obj or enum types holding "const" types.
  • The or type can be more than one level deep.

json-size for MessagePack

• Implement fast approx. MessagePack size calculation of a JSON object.

Reactive-RPC NDJSON

Expose Reactive-RPC server using HTTP streaming requests for uWebsocket.js server

• NDJSON
  • GET
  • POST

Time travel

Extend operations to contain information needed to undo that operation.

General

• Add ability to applyPatch function to automatically augment operations with data necessary for time travel.
• Add ability to remove applied operations, i.e. time travel.
• Set up fuzz testing.
  • JSON document is randomly generated.
  • Random operation patches are generated and applied.
  • Document is time travelled and each step is asserted agains the actual document.

JSON Patch

• add operation — add old key.
• remove operation — add old key.
• replace operation — add old key.
• move operation — add old key.
• copy operation — add old key.

JSON Patch Extended

• flip — investigate if changes needed.
• inc — investigate if changes needed.
• str_ins — investigate if changes needed.
• str_del — add old key .
• split — investigate if changes needed.
• merge — investigate if changes needed.
• extend — save old props.

`json-type` to TypeScript

• Implement function which converts json-type schema to TypeScript types.
• Maybe use AST builder to generate a proper AST?
  • Although it is not sensible to use it on the Web.
• Convert json-type to custom TypeScript AST.
• Output JSDoc comments.
• Generate TypeScript for services.
• Generate TypeScript for RPC methods.

Sub-collections

• Add ability for collections to have sub-collections.
• Sub-collections could be nested multiple levels deep.
• Sub-collections should have their own schema.

Binary encoding for JSON Patch