pascaldekloe/colfer

binary serialization format

colfer's Issues

Comments in generated code

Hi,

colfer place the name of schema file to the comment it generates via Maven Plugin, I am using a Windows machine and colfer does not escape the "" in file path and java compiler then complains about 'illegal escape character', is there a way to disable this comment or let it the colfer escape it before putting in the comment?

Thx

Try one allocation when unmarshalling multiple strings

Can you support the Object-C language?

Can you support the Object-C language?

[JAVA] NPE if I try to serialize an object with a null property

During the marshalling this part throws the NPE

if (!this.property.isEmpty()) {

Integer overflow in generated Go code on 32 bit

Hi,
I used 'go get' to get the Colfer tool yesterday and tried today to generate a simple schema. Unfortunately the generated code does not compile under x86 32-bit due to uint overflow errors. Specifically the timestamp causes the issue:
if v := o.From; !v.IsZero() { if s := uint(v.Unix()); s < 1<<32 { l += 9 } else { l += 13 } }

Colfer.go:160: constant 4294967296 overflows uint

I'm running go1.7.3 linux/386 if that helps. I've attached the .colf file and I'll gladly help test the fix!

Br.
Andreas

schema.colf.zip

Compiler generates non compiling code (Java)

I tried it with the downloaded compiler and this colf file:

package asdf

type PrimArraysColfer struct {
id []uint64
time []uint64
valueInteger []int32
valueInteger64 []int64
valueFloat []float32
valueDouble []float64
}

BufferOverflowException in writeObject

Maybe I missed something, but it looks like a bug:
In Java, in generated class there is writeObject(ObjectOutputStream out) method which uses buf array of size 1024 by default. In case of java.nio.BufferUnderflowException size of it is enlarged 4 times.
But in case when buf array is too small to handle marshalled object, java.nio.BufferOverflowException is thrown by marshal(byte[] buf, int offset) method, not java.nio.BufferUnderflowException, so buf is not enlarged and writeObject ends with exception.

No JavaScript Benchmarks?

Would love to see how the built-in JSON and Protobuf fare against colfer. Its kinda unfair to see C, Go, and Java make it to the benchmark games and JS stays at home. Could it be that the JS version couldn't live up to the Colfer performance promise?

EBNF Spec Needed

The current spec is ambiguous. e.g.

"Data structures consist of zero or more field value definitions followed by a termination byte 0x7f."

• Data structures: why the plural? Colfer is one data structure or consecutive data structures in a stream?
• the whole format is terminated by 0x7f or each of the field followed by 0x7f? (I guess the former)

Could you please write an EBNF spec? EBNF is unambiguous and uses less characters than descriptions. A good example is The Go Programming Language Specification.

Is this used in production anywhere?

Hi, I work for a large mobile app company that is interested in replacing JSON with something faster. The one disappointing thing is that colfer is written in Go and JS it looks like, which we can't use for iOS or Android (without a fair amount of finagling at least), so are there any plans for it to get ported to C/C++? Also, are there any companies that use it? It would be great to see a list of some companies that use or even their experiences, as that would really sell the format. How much security auditing has occurred?

The numbers on https://github.com/eishay/jvm-serializers/wiki are tantalizing, so I'm curious to hear more :)

[Feature request] publish colfer JS to npmjs.com

It'd be great to be able to install a colfer JS lib with npm.

Suggestion: Continuous Fuzzing

Hi, I'm Yevgeny Pats Founder of Fuzzit - Continuous fuzzing as a service platform.

I saw that you implemented Fuzz targets but they are currently not running as part of the CI.

We have a free plan for OSS and I would be happy to contribute a PR if that's interesting.
The PR will include the following

• Continuous Fuzzing of master branch which will generate new corpus and look for new crashes
• Regression on every PR that will run the fuzzers through all the generated corpus and fixed crashes from previous step. This will prevent new or old bugs from crippling into master.

You can see our basic example here and you can see an example of "in the wild" integration here.

Let me know if this is something worth working on.

Cheers,
Yevgeny

How use library for Java

Please help me i look a stats of library for read / write versus other, You have a good performance because i don't undersand how use this. Thank for read.

C++ support

Weird disprepancy in SizeMax sanity checks

Hey again,

While porting the code to Rust, I've come accross a very weird logic difference in two (functionally) identical sanity checks in generated Go code.

Here and here it checks if l is not overflowing ColferMaxSize, which seems logical and reasonable.

But a bit before (here) it performs the same check against x which is only the length of the list of nested objects.
Shouldn't it check against l instead like the others?

Maybe the generated test is not up to date and the bug has already been fixed, I have no idea, but in any case this might need a change.

Document uint16 better.

Why is uint16's compressed and uncompressed flipping the behavior of (index | 0x80)? This flag sort of is the anti-pattern of what uint32 and uint64 do. It's a bit confusing.

Integrate with Spring

How can we integrate with Spring Rest Controllers

Java Builder pattern for code chaining

currently all "setValue(Value x)" methods return void , if they would return the object then code-chaining would become possible.

example current:

SomeClass sc = new SomeClass();
sc.setValue(42);
sc.setStringValue("string");

example builder pattern:

SomeClass sc = new SomeClass().setValue(42).setStringValue("string");

Field order optimizations

The order for struct equality could be improved. Also for C and Go the ordering of the fields is relevant (memory alignment).

Support fixed-size arrays

For most languages this prevents memory allocation as the content can be embedded into the struct. There are many good uses such as IPv6 addresses, UUIDs and binary signatures.

Replace JavaScript with WASM

Poor performance as seen at #37 seems fixable with WASM.

How to set a longer package name in Java?

I tried different things. How can I set the package name to e.g. "com.cedarsoft.test.colfer"?

why not making colf java based on annotation processing

@Colfer could be used to identify pojo classes. Then the annotation processor would scan the class and generate the code for serialization / deserialization.

This is the most popular java way to do it. Annotation processors are a standard part of the java compiler.

Colf has a great potential on Android. And annotation processors are very common in Android builds.

Custom Default Value

The schema could specify other defaults than the zero value. @magiconair

Document limitation: Only 126 fields, ever, no removing

I wish it had been documented that a colf type cannot have more than 126 fields, ever, and this includes "deleted" fields.

(Why 126? "Each definition starts with an 8-bit header. The 7 least significant bits identify the field by its (0-based position) index in the schema." and 127 is reserved for the termination byte.)

Related, I wish https://github.com/pascaldekloe/colfer/#compatibility had explicitly said "Fields can never be removed."

Im interested in using colfer for a project... how stable is this project?

I am building a big/important project and I need a stable option for serialization. I am considering Msgpack and Protobuf as alternatives but Colfer has a nice schema and seems to be more performant. How stable/maintained is this project, would you say its production ready?

Thanks!

Support re-use of existing arrays if of same size.

This is Java-specific, I'm not sure how other generations work. When unmarshalling arrays (any list or byte array) a new array is always constructed. This needlessly creates garbage when the old array has the exact same size as the new array and can be directly re-used. This can be a common use-case for some applications (fixed size images or buffers, for example).

Rust support

Hello,

I wanted to ask about rust support.

Considering how such code is usually generated with macros, should we just create a crate that just implements the protocol/ser/deser logic?
Also, conforming with Serde to get automatic ser/deser of rust struct would be a very nice addition as well.

Cheers,

Mathieu

not real problem.

would you please refer golang/go#13560

Offer serial size limit constants when possible

For data structures without text, binaries and lists the serial size limit is constant.

Clarification request for "Robust against malicious input"

The Features section on the project page mentions that this serialization format is "Robust against malicious input".

Could it be a little more detailed than that? To what degree and how this robustness is achieved?

Problem with int64 type in javascript API

We have a problem: when maven generate javascript api our parameters with int64 type have equals and incorrect index. Please fix it, as soon as possible please, it is very important for us.

In example serverEventDatetime parameter is int64.

this.Journal.prototype.marshal = function () {
    var segs = [];
    if (this.serverEventDatetime) {


        var seg = [4]; // PROBLEM IN THIS LINE, MUST BE: var seg = [0];


        if (this.serverEventDatetime < 0) {
            seg[0] |= 128;
            if (this.serverEventDatetime < Number.MIN_SAFE_INTEGER)
                fail('colfer: api/Journal field serverEventDatetime exceeds Number.MIN_SAFE_INTEGER');
            encodeVarint(seg, -this.serverEventDatetime);
        } else {
            if (this.serverEventDatetime > Number.MAX_SAFE_INTEGER)
                fail('colfer: api/Journal field serverEventDatetime exceeds Number.MAX_SAFE_INTEGER');
            encodeVarint(seg, this.serverEventDatetime);
        }
        segs.push(seg);
    }
    if (this.subsystemCode) {
        var utf = encodeUTF8(this.subsystemCode);
        var seg = [1];
        encodeVarint(seg, utf.length);
        segs.push(seg);
        segs.push(utf)
    }
    if (this.code) {
        var utf = encodeUTF8(this.code);
        var seg = [2];
        encodeVarint(seg, utf.length);
        segs.push(seg);
        segs.push(utf)
    }
// ................................................................................

this.Journal.prototype.unmarshal = function (data) {
// ................................................................................
//IN UNMARSHAL HEADER CORRECT
    if (header == 0) {
        var x = readVarint();
        if (x < 0) fail('colfer: api/IJournal field serverEventDatetime exceeds Number.MAX_SAFE_INTEGER');
        this.serverEventDatetime = x;
        readHeader();
    } else if (header == (0 | 128)) {
        var x = readVarint();
        if (x < 0) fail('colfer: api/Journal field serverEventDatetime exceeds Number.MAX_SAFE_INTEGER');
        this.serverEventDatetime = -1 * x;
        readHeader();
    }

    if (header == 1) {
        var size = readVarint();
        if (size < 0)
            fail('colfer: api.Journal.subsystemCode size exceeds Number.MAX_SAFE_INTEGER');
        else if (size > colferSizeMax)
            fail('colfer: api.Journal.subsystemCode size ' + size + ' exceeds ' + colferSizeMax + ' UTF-8 bytes');

        var start = i;
        i += size;
        if (i > data.length) fail(EOF);
        this.subsystemCode = decodeUTF8(data.subarray(start, i));
        readHeader();
    }

    if (header == 2) {
        var size = readVarint();
        if (size < 0)
            fail('colfer: api.Journal.code size exceeds Number.MAX_SAFE_INTEGER');
        else if (size > colferSizeMax)
            fail('colfer: api.Journal.code size ' + size + ' exceeds ' + colferSizeMax + ' UTF-8 bytes');

        var start = i;
        i += size;
        if (i > data.length) fail(EOF);
        this.code = decodeUTF8(data.subarray(start, i));
        readHeader();
    }
// ................................................................................

Support reading/writing to/from Input/OutputStreams

Seems to be more convenient for the Java user.

Support of "oneof" enum based types

Protobuf allows creation of Messages which can hold one element of a certain set.

message Message {
    oneof value {  // <-- creates field and enum with {A, B}
        MsgA A = 1;
        MsgB B = 2;
    }
}
Message MsgA {
   ....
}
Message MsgB {
   ....
}

which allows a switch over the generated enum-field:

Message m = Message.parse(inputstream);
switch(m.getValueCase())
{
    case A: m.getA(); ...
    case B: m.getB(); ...
}

which is good, because java checkstyles can issue warnings when not all enum fields are checked in a switch-statement

from my understanding colfer supports this by doing something like:

type Message struct {
	head text
	body binary
}
type A struct {
...
}

and by constructing java objects like this:

A a = new A(); a.set(....)....;
byte[] b = new byte[1024];
a.marshal(b,0);
Message m = new Message(); m.setHead(A.class.getSimpleName()); m.setBody(b)

which in return allows switching over the class-string

switch(t.getHead()) {
 case "A": new A().unmarschal(t.getBody())
}

but i find the creation of these java object a little cumbersome, just so i can easily switch over them

JS version works wrong

Hi. I have a problem with generated JS code. I use Golang as a server and JS as client. Golang version makes valid marshal and unmarshal. JS version returns wrong packet after unmarshalling of packet from Golang server.
It's my code:

type update struct {
	id uint16
	x float32
	y float32
	rotation float32
	DT float32
	HP uint8
	commandId uint32
	maxBulletId uint16
}

type update_arr struct {
	packetType uint8
	players []update
}

Exactly update_arr packet works wrong in JS version.

On Golang server I have this data:

{
  "id": 0,
  "x": 100,
  "y": 100,
  "rotation": 0,
  "DT": 0,
  "HP": 100,
  "commandId": 0,
  "maxBulletId": 0
}

But on JS client:

{
  "id": 0,
  "x": 2.4262940954800533e35,
  "y": -131072.03125,
  "rotation": 0,
  "DT": 0,
  "HP": 100,
  "commandId": 0,
  "maxBulletId": 0
}

Will colfer support uint32/uint64 list in the future?

I believe uint32/uint64 list is quite common and useful, will colfer support it in the future?

Support for Python

Hi, is there any chance for Python support?

Timezone offset for Go

It would be great if this could be supported. Right now I have to serialise extra fields to rematerialize it correctly.

I think, unless it is very hard, that I can give it a stab if you point me in the right direction.

Simple usage example

I am interested in seeing a simple Java code example that uses Colfer for serialization into a ByteStream. I am currently conducting a study on the energy efficiency of serialization/deserialization on Android devices, specifically with regards to large numerical data types such as vectors, and will like to include Colfer in my study. Short of asking @eishay, and because it will provide a quick head-start, a link to a very basic use case would be appreciated.

Offer forward compatibility

Unmarshalling can skip unknown fields / headers when the binary's size is known.
-- @awalterschulze

Question: why not just use a subset of Go as the IDL?

The IDL is so similar to Go. Why not just use a subset of Go directly? e.g.

• time.Time for timestamp
• []byte for binary

Suggestion:add the -o parameter and specify the output file name.

Lists for integers

What about maps ?

It would be nice to serialize maps, maybe as a simple 2D array ?

Backwards compatibility with version 1

The generated code may include code for parsing the version 1 format. To distinguish, version 2 could start with the reserved header byte value 255.

Segmentation fault in a unique case

Hi,
I hope all are great.

Scenario:
I have the following mapping in my chappee.colf file.

package chappee
type mappee {
maptype uint32
msgtype uint32
callmsg text
debugleve text
minseed uint64
maxseed uint64
}

During packing, if I set the last two values, the packing is successful.
Now, if during packing I pack first 4 (or any of them) and don't pack the last 2, I get a segmentation fault.
I think it would be great that Colfer itself should take care of the values which are not set.

It can be taken as a bug, or it can be taken as a feature request.

Cheers,
infoginx.com

Use TypedArray on numeric lists for JavaScript

Java String fields when explicitly set to null throws NullPointerException when marshal() is called

I am using the latest (1.11.2) colfer maven plugin to generate Java objects for serialization and I noticed that when String fields are explicitly set to null, a NullPointerException is thrown. It looks like the generated code is generating the following code for String name title:

if (! this.title.isEmpty()) {

Can this be changed to check for null first before calling isEmpty()? Something like
if (this.title!=null && ! this.title.isEmpty()) {

partial unmarshal java

Hello,

A question. I have my serialized colfer byte array inside a ByteBuffer. And i would like to access just some specific fields. Is this possible without loading everything or i should write a customised version of generated unmarshal method?

Regards
Tamer

[JAVA] NPE while comparing two object with .equals

This is the equals method of my class (compiled with colfer):

public final boolean equals(Metadata o) {
    return o != null 
        && o.getClass() == Metadata.class 
        && this.creationTime == null ? o.creationTime == null : this.creationTime.equals(o.creationTime) 
        && this.statistic == null ? o.statistic == null : this.statistic.equals(o.statistic) 
        && this.source == null ? o.source == null : this.source.equals(o.source) 
        && this.agent == null ? o.agent == null : this.agent.equals(o.agent) 
        && this.accessControl == null ? o.accessControl == null : this.accessControl.equals(o.accessControl);
  }

If o == null i get a NPE at the "creationTime" line. From my understanding the ? condition are evaluated before testing the && condition. So in this method this.creationTime.equals triggers a NPE.

I tried adding brackets to see if i could have the VM to evaluate the null check before all the others and it does work:

public final boolean equals(Metadata o) {
    return o != null 
        && o.getClass() == Metadata.class 
        && (this.creationTime == null ? o.creationTime == null : this.creationTime.equals(o.creationTime))
        && (this.statistic == null ? o.statistic == null : this.statistic.equals(o.statistic))
        && (this.source == null ? o.source == null : this.source.equals(o.source))
        && (this.agent == null ? o.agent == null : this.agent.equals(o.agent))
        && (this.accessControl == null ? o.accessControl == null : this.accessControl.equals(o.accessControl));
  }