sarvalabs/js-polo

JavaScript Implementation of the POLO Serialization Format

js-polo

js-polo is a JS/TS implementation of the POLO encoding and object serialization scheme. POLO stands for Prefix Ordered Lookup Offsets.

It is intended for use in projects that prioritize deterministic serialization, minimal wire sizes and serialization safety. POLO follows a very strict specification that is optimized for partial decoding and differential messaging. This implementation is compliant with the POLO Specification that describes both the encoding (wire) format as well as implementation guidelines for several languages.

Installation

Install the latest release using the following command.

npm install js-polo

Features

Deterministic Serialization

POLO's strict specification is intended to create the same serialized wire for an object regardless of implementation. This is critical for cryptographic security with operations such as hashing which is used to guarantee data consistency and tamper proofing.

High Wire Efficiency

POLO has a highly optimized wire format allows messages to be relatively small, even surpassing Protocol Buffers occassionaly. This is mainly because it supports a larger type based wire tagging that allows some information (especially metadata) to be passed around inferentially and thus reducing the total amount of information actually present in the wire.

This is augmented by the fact that POLO supports Atomic Encoding, because of which simple objects such as integers & strings can be encoded without needing to be wrapped within a larger (structural) message

Partial Encoding/Decoding Constructs

The POLO wire format prefixes all metadata for the wire in the front, this metadata includes the wire type information as well as the offset position of the data. This kind of lookup based offset tags allows us to directly access the data for a particular field order. This capability is currently supported using the Document and Raw constructs.

The Document construct is used to create a string indexed collection of Raw object and it collapses into a special form of the POLO wire called the document-encoded wire with the WireDoc wire type. It's unique in that it preserves the field name (string index) for the data unlike regular POLO encoding and is similar to encoding schemes like JSON or YAML and consequently consumes more wire space.

The Raw construct is useful for capturing the wire data for a specific field in the struct. It can also be used to define a structure that 'skips' the unrequired fields by capturing their raw wire instead of decoding them.

Partial encodign/decoding capabilities for this implementation are unfinished can be extended to support field order based access/write for regular POLO encoded wires in the future.

Differential Messaging (Coming Soon)

POLO's partially encoding and field order based indexing (and string based indexing for document encoded wires) allows the possibility for messaging that only allows the transmission of the difference between two states, this is useful for any version managment system where the same data is incrementally updated and transmitted, the ability to index the difference and only transmit the difference can result in massive reduction in the wire sizes for these use cases that often re-transmit already available information.

Examples

Polorizer

const fruit = {
    name: 'orange',
    cost: 300,
    alias: ['tangerine', 'mandarin']
};

const schema = {
    kind: 'struct',
    fields: {
        name: {
            kind: 'string'
        },
        cost: {
            kind: 'integer'
        },
        alias: {
            kind: 'array',
            fields: {
                values: {
                    kind: 'string',
                }
            }
        }
    }
}

const polorizer = new Polorizer();
polorizer.polorize(fruit, schema);
console.log(polorizer.bytes())

// Output:
/* 
    [
        14, 79, 6, 99, 142, 1, 111, 114, 97, 110, 103, 101, 1, 44, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114, 105, 110, 101, 109,  97, 110, 100,  97, 114, 105, 110
    ]
*/

Depolorizer

const wire = new Uint8Array([14, 79, 6, 99, 142, 1, 111, 114, 97, 110, 103, 101, 1, 44, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114, 105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110])
const schema = {
    kind: 'struct',
    fields: {
        name: {
            kind: 'string'
        },
        cost: {
            kind: 'integer'
        },
        alias: {
            kind: 'array',
            fields: {
                values: {
                    kind: 'string',
                }
            }
        }
    }
}

const depolorizer = new Depolorizer(wire)
console.log(depolorizer.depolorize(schema))

// Output:
/* 
    { 
        name: 'orange', 
        cost: 300, 
        alias: [ 'tangerine', 'mandarin' ] 
    }
*/

Document Encoding

// Create a Fruit object
const fruit = {
    name: 'orange',
    cost: 300,
    alias: ['tangerine', 'mandarin']
};

const schema = {
    kind: 'struct',
    fields: { 
        name: { kind: 'string' },
        cost: { kind: 'integer' },
        alias: { 
            kind: 'array', 
            fields: { 
                values: { kind: 'string' } 
            } 
        }
    }
};

// Encode the object into a Document
const document = documentEncode(fruit, schema);

console.log(document.getData());
console.log(document.bytes());

// Output:
/*
    {
        name: Raw {
            bytes: Uint8Array(7) [
                6, 111, 114, 97, 110, 103, 101
            ]
        },
        cost: Raw { bytes: Uint8Array(3) [ 3, 1, 44 ] },
        alias: Raw {
            bytes: Uint8Array(22) [
                14, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114, 105, 110, 101, 109,  97, 110, 100, 97, 114, 105, 110
            ]
        }
    }
*/

/*
    [
        13, 175, 1, 6, 85, 182, 3, 245, 3, 166, 4,229, 4, 97, 108, 105, 97, 115, 14, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114, 105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110, 99, 111, 115, 116, 3, 1, 44, 110, 97, 109, 101, 6, 111, 114, 97,110, 103, 101
    ]
*/

Document Decoding

Decode Document

const wire = new Uint8Array([
    13, 175, 1, 6, 85, 182, 3, 245, 3, 166, 4, 229, 4, 97, 108, 105, 
    97, 115, 14, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114, 105, 110, 
    101, 109, 97, 110, 100, 97, 114, 105, 110, 99, 111, 115, 116,
    3, 1, 44, 110, 97, 109, 101, 6, 111, 114,  97, 110, 103, 101
]);

const schema = {
    kind: 'struct',
    fields: { 
        name: { kind: 'string' },
        cost: { kind: 'integer' },
        alias: { 
            kind: 'array', 
            fields: { 
                values: { kind: 'string' } 
            } 
        }
    }
};

const document = new Document(wire, schema);

console.log(document.getData());

// Output:
/*
    {
        name: Uint8Array(7) [
            6, 111, 114, 97,
        110, 103, 101
        ],
        cost: Uint8Array(3) [ 3, 1, 44 ],
        alias: Uint8Array(22) [
            14,  63,   6, 150,   1, 116,
            97, 110, 103, 101, 114, 105,
        110, 101, 109,  97, 110, 100,
            97, 114, 105, 110
        ],
    }
*/

Decode Struct

const wire = new Uint8Array([
    13, 175, 1, 6, 85, 182, 3, 245, 3, 166, 4, 229, 4, 97, 108, 105, 97, 115, 14, 63, 6, 150, 1, 116, 97, 110, 103, 101, 114, 105, 110, 101, 109, 97, 110, 100, 97, 114, 105, 110, 99, 111, 115, 116, 3, 1, 44, 110, 97, 109, 101, 6, 111, 114,  97, 110, 103, 101
]);

const schema = {
    kind: 'struct',
    fields: { 
        name: { kind: 'string' },
        cost: { kind: 'integer' },
        alias: { 
            kind: 'array', 
            fields: { 
                values: { kind: 'string' } 
            } 
        }
    }
};

const depolorizer = new Depolorizer(wire);

console.log(depolorizer.depolorize(schema));

// Output:
/* 
    { 
        name: 'orange', 
        cost: 300, 
        alias: [ 'tangerine', 'mandarin' ] 
    }
*/

Contributing

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as below, without any additional terms or conditions.

License

© 2023 Sarva Labs Inc. & MOI Protocol Developers.

This project is licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

The SPDX license identifier for this project is MIT OR Apache-2.0.

js-polo's Issues

Simple Encode & Decode Functionality

Simple POLO Encoding/Decoding

• Implement the Polorizer class for simple serialization into POLO data
• Implement the Depolorizer class for simple deserialization from POLO data.

Polorizer Methods

// Polorizer internally uses the WriteBuffer class and its methods

polorizeRaw(value: Raw) Uint8Array

polorizeBool(value: Boolean) Uint8Array

polorizeInteger(value: Number|BigInt) Uint8Array

polorizeFloat(value: Number) Uint8Array

polorizeString(value: String) Uint8Array

polorizeBytes(value: Uint8Array) Uint8Array

Depolorizer Methods

// Depolorizer internally uses the ReadBuffer class and its methods

depolorizeRaw(data: Uint8Array) Raw

depolorizeBool(data: Uint8Array) Boolean

depolorizeInteger(data: Uint8Array) Number|BigInt

depolorizeFloat(data: Uint8Array) Number

depolorizeString(data: Uint8Array) String

depolorizeBytes(value: Uint8Array) Uint8Array

Fix Non Primitive Depolorizer Methods

Description

In the current implementation of the depolorizeArray, depolorizeMap, and depolorizeStruct methods, there is a missing handler for the WIRE_NULL case. Consequently, when the wire type is WIRE_NULL, these methods return an error. However, such behavior contradicts the expected functionality, as there should be explicit handling for the WIRE_NULL case.

Expected Behavior:

Each of the mentioned methods should explicitly manage the scenario where the wire type is WIRE_NULL. This could involve returning an appropriate value based on the method's return type (an empty array [] for arrays, a new map Map() for maps, and an empty object {} for structs).

Invalid Output While Depolarizing Bytes to Boolean Array

Invalid Output While Depolarizing Bytes to Boolean Array

When depolarizing bytes using the provided schema and polarization method, the resulting data differs from the originally polarized data, leading to unexpected output.

Encoding Schema:

const schema = {
    kind: "array",
    fields: {
        values: {
            kind: "bool",
        },
    },
};

Polorization Example:

const arr = [true, false, true, false, true, false, true, false, true, false, true, false, true, false, true, false];
const polorizer = new Polorizer();
polorizer.polorize(arr, schema);
console.log(polorizer.bytes());
// Output: Uint8Array(19) [ 14, 143, 2, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1, 2, 1 ]

The hex representation of the polarized bytes is 0e8f0202010201020102010201020102010201.

Depolarization Approach:

const data = new Depolorizer(polorizer.bytes()).depolorize(schema);
console.log(data);
// Output: [ true, true, true, true, true, true, true, true ]

Expected Behavior:

The depolarized data should match the original array used for polarization, maintaining the correct boolean values.

Actual Behavior:

The depolarized data consists of a sequence of true values, which is inconsistent with the original polarized array.

Environment Details

• Language: Typescript
• Js-Polo Version: v0.1.3
• Node.JS Version: v21.7.0

Schema Based Encode/Decode

Schema Encoding & Decoding

To support safe encoding of compound types like arrays and objects, we need the user to either use Pack encoding with Packer and Unpacker. An alternative model is to allow the user to provide a type schema that specifies how to serialize the properties/elements of a compound type.

This schema can be provided to the special methods on Polorizer, Depolorizer, Packer, Unpacker and Document as follows:

Polorizer.polorizeAs(value: Any, schema: Schema) Uint8Array

Depolorizer.depolorizeAs(data: Uint8Array, schema: Schema) Any

Packer.packAs(value: Any, schema: Schema)

Unpacker.unpackAs(schema: Schema) Any

Document.getAs(key: String, schema: Schema) Any

Document.setAs(key: String, value: Any, schema: Schema) 

DocumentDecode(data: Uint8Array, schema: Schema)

Schema Definition

The schema is defined in JSON as follows:

// Schema Object
{
    "kind": <str>,
    "fields": <map; string -> Schema>
}

• The valid type strings are null, raw, bytes, bool, integer, float, string with compound ones being array, map and struct.
• The kind is a mandatory parameter and must be a valid type string. If the kind is non-compound, then fields is not required.
• If the kind is array, then fields must contain a property values which has a valid Schema representing the type of the array element. (Expects array elements to be the same type)
• If the kind is map, then fields must contain the properties keys and values, each of which must have a valid Schema, representing the map key and value types respectively.
• If the kind is struct, then the properties of fields represent the type for each field. This can be done with numeric values such as "0" and "1" which represent the field order for pack-encoded data or with string keys like "name" and "age" for document-encoded data. (numeric properties will be considered valid keys for document-encoded wires while the string properties are not allowed for pack-encoded data). Each property must contain a Schema representing the type for that field/property in the object

Fix Integer Size Bug

Description

A bug in the intSize() method causes incorrect byte size calculations for certain integer inputs. Specifically, an edge case where the input number equals 256 results in an incorrect byte size being returned.

Expected Behavior:

The intSize() method should accurately calculate the byte size for all integer inputs, including edge cases.

Steps to Reproduce:

1. Call the intSize() method with an input value of 256.
2. Observe the returned byte size, which should be 2 bytes, but is currently returning 1 byte.

Pack Encoding with Packer & Unpacker

POLO Pack Encoding

• Implement the Packer class that provides an interface for continous serialization into pack encoded data
• Implement the Unpacker class that provides an interface for continous deserialization from pack encoded data
• Packer and Unpacker wrap around WriteBuffer and LoadReader classes respectively and expose methods similar to Polorizer and Depolorizer

Document Encoding

POLO Document Encoding

• Implement a Document class that holds string indexed Raw objects.
• Document constructor must allow for it to create a new empty empty but also an existing object for a Uint8Array that represents a document-encoded wire.
• Document must expose methods to Get_ and Set_ various types based on string key. The GetRaw method must return the Raw object for that key and similarly SetRaw will insert a Raw object for a given key.
• Document has a method Size() Number to count the number of elements
• DocumentEncode is a function that accepts any arbitrary object and encodes its properties into a Document.

Fix buffer not defined bug

Description

The browser environment throws a "buffer not defined" error due to the usage of the "buffer" data type, which is not available in the browser's runtime. This bug hinders the proper functioning of the package when running in a browser.

POLO I/O Buffers Implementation

POLO I/O Buffers

Implement the following classes for I/O Manipulation.
All of these classes must be unexported and only used internally.

• WireType enum and Wire class for wire type methods
• Raw for holding raw encoded data.
• ReadBuffer and ByteReader for wire reading.
• WriteBuffer for wire writing.
• LoadReader for sequentially unpacking WireLoad

WriteBuffer Methods

// Writes the data in the Buffer tagged with the wiretype to the write buffer
write(wire: WireType, data: Uint8Array)

// Returns the write buffer contents as bytes
bytes() Uint8Array

// Returns the write buffer wrapped with a WireLoad tag as bytes
load() Uint8Array
 
// Writes an empty tag of WireNull into head of the write buffer
writeNull() 

// Writes the value to the write buffer with the WireRaw tag
writeRaw(value: Raw)

// Writes the value as a WireTrue or WireFalse wire to the write buffer
writeBool(value: Boolean)

// Writes the value (must be Integer) as a WirePosInt or WireNegInt wire to the write buffer
writeInteger(value: Number|BigInt) 

// Writes the value as a WireFloat wire to the write buffer
writeFloat(value: Number)

// Writes the value as a WireWord wire to the write buffer
writeString(value: String)

// Writes the value as a WireWord wire to the write buffer
writeBytes(value: Uint8Array) 

ReadBuffer Methods

// Returns the read buffer as bytes
bytes() Uint8Array

// Attempts to convert the read buffer into a load reader
load() LoadReader

// Reads the data in the read buffer into raw bytes
readRaw() Raw

// Reads the data in the read buffer into a boolean
readBool() Boolean

// Reads the data in the read buffer into an integer
readInteger() Number|BigInt

// Reads the data in the read buffer into an float
readFloat() Number

// Reads the data in the read buffer into an string
readString() String

// Reads the data in the read buffer into bytes
readBytes() Uint8Array

LoadReader Methods

// Returns whether all data in the load read has been read
done() Boolean

// Returns the next element in the load as a ReadBuffer
next() ReadBuffer