mailru/easyjson

Fast JSON serializer for golang.

easyjson

Package easyjson provides a fast and easy way to marshal/unmarshal Go structs to/from JSON without the use of reflection. In performance tests, easyjson outperforms the standard encoding/json package by a factor of 4-5x, and other JSON encoding packages by a factor of 2-3x.

easyjson aims to keep generated Go code simple enough so that it can be easily optimized or fixed. Another goal is to provide users with the ability to customize the generated code by providing options not available with the standard encoding/json package, such as generating "snake_case" names or enabling omitempty behavior by default.

Usage

Install:

# for Go < 1.17
go get -u github.com/mailru/easyjson/...

or

# for Go >= 1.17
go get github.com/mailru/easyjson && go install github.com/mailru/easyjson/...@latest

Run:

easyjson -all <file>.go

The above will generate <file>_easyjson.go containing the appropriate marshaler and unmarshaler funcs for all structs contained in <file>.go.

Please note that easyjson requires a full Go build environment and the GOPATH environment variable to be set. This is because easyjson code generation invokes go run on a temporary file (an approach to code generation borrowed from ffjson).

Serialize

someStruct := &SomeStruct{Field1: "val1", Field2: "val2"}
rawBytes, err := easyjson.Marshal(someStruct)

Deserialize

someStruct := &SomeStruct{}
err := easyjson.Unmarshal(rawBytes, someStruct)

Please see the GoDoc for more information and features.

Options

Usage of easyjson:
  -all
    	generate marshaler/unmarshalers for all structs in a file
  -build_tags string
        build tags to add to generated file
  -gen_build_flags string
        build flags when running the generator while bootstrapping
  -byte
        use simple bytes instead of Base64Bytes for slice of bytes
  -leave_temps
    	do not delete temporary files
  -no_std_marshalers
    	don't generate MarshalJSON/UnmarshalJSON funcs
  -noformat
    	do not run 'gofmt -w' on output file
  -omit_empty
    	omit empty fields by default
  -output_filename string
    	specify the filename of the output
  -pkg
    	process the whole package instead of just the given file
  -snake_case
    	use snake_case names instead of CamelCase by default
  -lower_camel_case
        use lowerCamelCase instead of CamelCase by default
  -stubs
    	only generate stubs for marshaler/unmarshaler funcs
  -disallow_unknown_fields
        return error if some unknown field in json appeared
  -disable_members_unescape
        disable unescaping of \uXXXX string sequences in member names

Using -all will generate marshalers/unmarshalers for all Go structs in the file excluding those structs whose preceding comment starts with easyjson:skip. For example:

//easyjson:skip
type A struct {}

If -all is not provided, then only those structs whose preceding comment starts with easyjson:json will have marshalers/unmarshalers generated. For example:

//easyjson:json
type A struct {}

Additional option notes:

• -snake_case tells easyjson to generate snake_case field names by default (unless overridden by a field tag). The CamelCase to snake_case conversion algorithm should work in most cases (ie, HTTPVersion will be converted to "http_version").

• -build_tags will add the specified build tags to generated Go sources.

• -gen_build_flags will execute the easyjson bootstapping code to launch the actual generator command with provided flags. Multiple arguments should be separated by space e.g. -gen_build_flags="-mod=mod -x".

Structure json tag options

Besides standard json tag options like 'omitempty' the following are supported:

• 'nocopy' - disables allocation and copying of string values, making them refer to original json buffer memory. This works great for short lived objects which are not hold in memory after decoding and immediate usage. Note if string requires unescaping it will be processed as normally.
• 'intern' - string "interning" (deduplication) to save memory when the very same string dictionary values are often met all over the structure. See below for more details.

Generated Marshaler/Unmarshaler Funcs

For Go struct types, easyjson generates the funcs MarshalEasyJSON / UnmarshalEasyJSON for marshaling/unmarshaling JSON. In turn, these satisfy the easyjson.Marshaler and easyjson.Unmarshaler interfaces and when used in conjunction with easyjson.Marshal / easyjson.Unmarshal avoid unnecessary reflection / type assertions during marshaling/unmarshaling to/from JSON for Go structs.

easyjson also generates MarshalJSON and UnmarshalJSON funcs for Go struct types compatible with the standard json.Marshaler and json.Unmarshaler interfaces. Please be aware that using the standard json.Marshal / json.Unmarshal for marshaling/unmarshaling will incur a significant performance penalty when compared to using easyjson.Marshal / easyjson.Unmarshal.

Additionally, easyjson exposes utility funcs that use the MarshalEasyJSON and UnmarshalEasyJSON for marshaling/unmarshaling to and from standard readers and writers. For example, easyjson provides easyjson.MarshalToHTTPResponseWriter which marshals to the standard http.ResponseWriter. Please see the GoDoc listing for the full listing of utility funcs that are available.

Controlling easyjson Marshaling and Unmarshaling Behavior

Go types can provide their own MarshalEasyJSON and UnmarshalEasyJSON funcs that satisfy the easyjson.Marshaler / easyjson.Unmarshaler interfaces. These will be used by easyjson.Marshal and easyjson.Unmarshal when defined for a Go type.

Go types can also satisfy the easyjson.Optional interface, which allows the type to define its own omitempty logic.

Type Wrappers

easyjson provides additional type wrappers defined in the easyjson/opt package. These wrap the standard Go primitives and in turn satisfy the easyjson interfaces.

The easyjson/opt type wrappers are useful when needing to distinguish between a missing value and/or when needing to specifying a default value. Type wrappers allow easyjson to avoid additional pointers and heap allocations and can significantly increase performance when used properly.

Memory Pooling

easyjson uses a buffer pool that allocates data in increasing chunks from 128 to 32768 bytes. Chunks of 512 bytes and larger will be reused with the help of sync.Pool. The maximum size of a chunk is bounded to reduce redundant memory allocation and to allow larger reusable buffers.

easyjson's custom allocation buffer pool is defined in the easyjson/buffer package, and the default behavior pool behavior can be modified (if necessary) through a call to buffer.Init() prior to any marshaling or unmarshaling. Please see the GoDoc listing for more information.

String interning

During unmarshaling, string field values can be optionally interned to reduce memory allocations and usage by deduplicating strings in memory, at the expense of slightly increased CPU usage.

This will work effectively only for string fields being decoded that have frequently the same value (e.g. if you have a string field that can only assume a small number of possible values).

To enable string interning, add the intern keyword tag to your json tag on string fields, e.g.:

type Foo struct {
  UUID  string `json:"uuid"`         // will not be interned during unmarshaling
  State string `json:"state,intern"` // will be interned during unmarshaling
}

Issues, Notes, and Limitations

• easyjson is still early in its development. As such, there are likely to be bugs and missing features when compared to encoding/json. In the case of a missing feature or bug, please create a GitHub issue. Pull requests are welcome!

• Unlike encoding/json, object keys are case-sensitive. Case-insensitive matching is not currently provided due to the significant performance hit when doing case-insensitive key matching. In the future, case-insensitive object key matching may be provided via an option to the generator.

• easyjson makes use of unsafe, which simplifies the code and provides significant performance benefits by allowing no-copy conversion from []byte to string. That said, unsafe is used only when unmarshaling and parsing JSON, and any unsafe operations / memory allocations done will be safely deallocated by easyjson. Set the build tag easyjson_nounsafe to compile it without unsafe.

• easyjson is compatible with Google App Engine. The appengine build tag (set by App Engine's environment) will automatically disable the use of unsafe, which is not allowed in App Engine's Standard Environment. Note that the use with App Engine is still experimental.

• Floats are formatted using the default precision from Go's strconv package. As such, easyjson will not correctly handle high precision floats when marshaling/unmarshaling JSON. Note, however, that there are very few/limited uses where this behavior is not sufficient for general use. That said, a different package may be needed if precise marshaling/unmarshaling of high precision floats to/from JSON is required.

• While unmarshaling, the JSON parser does the minimal amount of work needed to skip over unmatching parens, and as such full validation is not done for the entire JSON value being unmarshaled/parsed.

• Currently there is no true streaming support for encoding/decoding as typically for many uses/protocols the final, marshaled length of the JSON needs to be known prior to sending the data. Currently this is not possible with easyjson's architecture.

• easyjson parser and codegen based on reflection, so it won't work on package main files, because they cant be imported by parser.

Benchmarks

Most benchmarks were done using the example 13kB example JSON (9k after eliminating whitespace). This example is similar to real-world data, is well-structured, and contains a healthy variety of different types, making it ideal for JSON serialization benchmarks.

Note:

• For small request benchmarks, an 80 byte portion of the above example was used.

• For large request marshaling benchmarks, a struct containing 50 regular samples was used, making a ~500kB output JSON.

• Benchmarks are showing the results of easyjson's default behaviour, which makes use of unsafe.

Benchmarks are available in the repository and can be run by invoking make.

easyjson vs. encoding/json

easyjson is roughly 5-6 times faster than the standard encoding/json for unmarshaling, and 3-4 times faster for non-concurrent marshaling. Concurrent marshaling is 6-7x faster if marshaling to a writer.

easyjson vs. ffjson

easyjson uses the same approach for JSON marshaling as ffjson, but takes a significantly different approach to lexing and parsing JSON during unmarshaling. This means easyjson is roughly 2-3x faster for unmarshaling and 1.5-2x faster for non-concurrent unmarshaling.

As of this writing, ffjson seems to have issues when used concurrently: specifically, large request pooling hurts ffjson's performance and causes scalability issues. These issues with ffjson can likely be fixed, but as of writing remain outstanding/known issues with ffjson.

easyjson and ffjson have similar performance for small requests, however easyjson outperforms ffjson by roughly 2-5x times for large requests when used with a writer.

easyjson vs. go/codec

go/codec provides compile-time helpers for JSON generation. In this case, helpers do not work like marshalers as they are encoding-independent.

easyjson is generally 2x faster than go/codec for non-concurrent benchmarks and about 3x faster for concurrent encoding (without marshaling to a writer).

In an attempt to measure marshaling performance of go/codec (as opposed to allocations/memcpy/writer interface invocations), a benchmark was done with resetting length of a byte slice rather than resetting the whole slice to nil. However, the optimization in this exact form may not be applicable in practice, since the memory is not freed between marshaling operations.

easyjson vs 'ujson' python module

ujson is using C code for parsing, so it is interesting to see how plain golang compares to that. It is important to note that the resulting object for python is slower to access, since the library parses JSON object into dictionaries.

easyjson is slightly faster for unmarshaling and 2-3x faster than ujson for marshaling.

Benchmark Results

ffjson results are from February 4th, 2016, using the latest ffjson and go1.6. go/codec results are from March 4th, 2016, using the latest go/codec and go1.6.

Unmarshaling

lib	json size	MB/s	allocs/op	B/op
standard	regular	22	218	10229
standard	small	9.7	14	720
easyjson	regular	125	128	9794
easyjson	small	67	3	128
ffjson	regular	66	141	9985
ffjson	small	17.6	10	488
codec	regular	55	434	19299
codec	small	29	7	336
ujson	regular	103	N/A	N/A

Marshaling, one goroutine.

lib	json size	MB/s	allocs/op	B/op
standard	regular	75	9	23256
standard	small	32	3	328
standard	large	80	17	1.2M
easyjson	regular	213	9	10260
easyjson*	regular	263	8	742
easyjson	small	125	1	128
easyjson	large	212	33	490k
easyjson*	large	262	25	2879
ffjson	regular	122	153	21340
ffjson**	regular	146	152	4897
ffjson	small	36	5	384
ffjson**	small	64	4	128
ffjson	large	134	7317	818k
ffjson**	large	125	7320	827k
codec	regular	80	17	33601
codec***	regular	108	9	1153
codec	small	42	3	304
codec***	small	56	1	48
codec	large	73	483	2.5M
codec***	large	103	451	66007
ujson	regular	92	N/A	N/A

* marshaling to a writer, ** using ffjson.Pool(), *** reusing output slice instead of resetting it to nil

Marshaling, concurrent.

lib	json size	MB/s	allocs/op	B/op
standard	regular	252	9	23257
standard	small	124	3	328
standard	large	289	17	1.2M
easyjson	regular	792	9	10597
easyjson*	regular	1748	8	779
easyjson	small	333	1	128
easyjson	large	718	36	548k
easyjson*	large	2134	25	4957
ffjson	regular	301	153	21629
ffjson**	regular	707	152	5148
ffjson	small	62	5	384
ffjson**	small	282	4	128
ffjson	large	438	7330	1.0M
ffjson**	large	131	7319	820k
codec	regular	183	17	33603
codec***	regular	671	9	1157
codec	small	147	3	304
codec***	small	299	1	48
codec	large	190	483	2.5M
codec***	large	752	451	77574

* marshaling to a writer, ** using ffjson.Pool(), *** reusing output slice instead of resetting it to nil

easyjson's Issues

Invalid code generated for slices of struct pointers

Easyjson adds unneeded and invalid package name to types when dealing with slices of struct pointers ([]*MyStruct).

Minimal example:

package issue

type S struct {
    B string
}

type Exp struct {
    A []*S
}

Relevant generated code:

func easyjson_decode_github_com_mailru_easyjson_issue_Exp(in *jlexer.Lexer, out *Exp) {
    in.Delim('{')
    for !in.IsDelim('}') {
        key := in.UnsafeString()
        in.WantColon()
        if in.IsNull() {
            in.Skip()
            in.WantComma()
            continue
        }
        switch key {
        case "A":
            in.Delim('[')
            if !in.IsDelim(']') {
                out.A = make([]*issue.S, 0, 8)
            } else {
                out.A = nil
            }
            for !in.IsDelim(']') {
                var v1 *issue.S
                if in.IsNull() {
                    in.Skip()
                    v1 = nil
                } else {
                    v1 = new(S)
                    (*v1).UnmarshalEasyJSON(in)
                }
                out.A = append(out.A, v1)
                in.WantComma()
            }
            in.Delim(']')
        default:
            in.SkipRecursive()
        }
        in.WantComma()
    }
    in.Delim('}')
}

Invalid code (issue is not imported and can't be since it would produce import cycle):

out.A = make([]*issue.S, 0, 8)

easyjson is not fully compatible with standard JSON in case of Marshaling

Consider a struct that contains an invalid marshal type, e.g. func:

type A struct {
  F func()
}

If we run json.Marshal(A{}), we should receive an *json.UnsupportedTypeError, but after easyjson generation, it becomes a *json.MarshalerError.

Relatively loose json type, numeric and string are compatible with each other

Migration from other languages to golang, can much some weakly typed languages even the Java string and numerical mutual compatibility and conversion, lead to the client without a fully the same json type, add a function in the generated code optional loose type model, used for compatibility problems left over by history

Failure in unmarshal surrogate unicode encoding (utf16) - BUG

Hi ,

Example JSON File is :

{
  "name":"Khashayar is not \ud83d\ude08"
}

And go model is :

type Test struct {
	Name string `json:"name"`
}

Unmarshal test instance on mentioned JSON file will fail with this error:
parse error: invalid character 'K' after top-level value near offset 12 of 'Khashayar is not \ud83d\ude08"
}

as it's obvious json file is valid and represent :

{"name":"Khashayar is not 😈"}

the problem is from processEscape in lexer.go .
i will send pull request soon .

Struct with no fields leads to the Divide by zero error

if some field of my struct has type "struct{}":

//easyjson:json
type MyStruct struct {
    Text string `json:"text"`
    EmptySlice []struct{} `json:"empty_slice"`
}

it leads to the following error, because count of fields is zero. If I add at least one field to this struct, then I don't have the error anymore:

user$ easyjson ./v1.go
panic: runtime error: integer divide by zero

goroutine 1 [running]:
panic(0x451d800, 0xc42000c0c0)
	/Users/user/.gobrew/versions/1.7.1/src/runtime/panic.go:500 +0x1a1
api/vendor/github.com/mailru/easyjson/gen.(*Generator).genTypeDecoderNoCheck(0xc4201e4780, 0x4848200, 0x44e9bc0, 0xc4202135c8, 0xb, 0x451c08f, 0x7, 0x0, 0x3, 0xc4202135cc, ...)
	/Users/user/go/src/api/vendor/github.com/mailru/easyjson/gen/decoder.go:99 +0x50fc
api/vendor/github.com/mailru/easyjson/gen.(*Generator).genTypeDecoder(0xc4201e4780, 0x4848200, 0x44e9bc0, 0xc4202135c8, 0xb, 0x451c08f, 0x7, 0x0, 0x3, 0x0, ...)
	/Users/user/go/src/api/vendor/github.com/mailru/easyjson/gen/decoder.go:68 +0x52c
api/vendor/github.com/mailru/easyjson/gen.(*Generator).genStructFieldDecoder(0xc4201e4780, 0x4848200, 0x456e180, 0x451c080, 0x7, 0x0, 0x0, 0x4848200, 0x44e9bc0, 0x451c089, ...)
	/Users/user/go/src/api/vendor/github.com/mailru/easyjson/gen/decoder.go:238 +0x27c
api/vendor/github.com/mailru/easyjson/gen.(*Generator).genStructDecoder(0xc4201e4780, 0x4848200, 0x456e180, 0x44f2360, 0xc4201b7e70)
	/Users/user/go/src/api/vendor/github.com/mailru/easyjson/gen/decoder.go:409 +0x131b
api/vendor/github.com/mailru/easyjson/gen.(*Generator).genDecoder(0xc4201e4780, 0x4848200, 0x456e180, 0xc420213e87, 0x0)
	/Users/user/go/src/api/vendor/github.com/mailru/easyjson/gen/decoder.go:333 +0xb3
api/vendor/github.com/mailru/easyjson/gen.(*Generator).Run(0xc4201e4780, 0x483daa0, 0xc42002c010, 0x0, 0x0)
	/Users/user/go/src/api/vendor/github.com/mailru/easyjson/gen/generator.go:182 +0x12e
main.main()
	/Users/user/go/src/api/api/handler/easyjson-bootstrap223403038.go:21 +0xd2
exit status 2
Bootstrap failed: exit status 1

In this situation I want to use empty struct to have just a fake empty slice to follow the api contract, but inside this struct I never need to pass any data, it just helps mobile apps to parse my response to the same structure we use in other handlers.

Please fix it. Thanks!

Package is a program, not an importable package

Fails when running against this file...

I've created package foo, and have the following file main.go

package main

import (
    "fmt"
)

type Foo struct {
    Name string
}

func main() {
    f := Foo{Name: "Bar"}
    fmt.Printf("%+v\n", f)
}

Output

$ easyjson -all main.go 
/var/folders/wj/ggsd2sgs2b1_vnplhkcv6g0jk5xz5m/T/easyjson715726581.go:8:3: import "github.com/scottjbarr/foo" is a program, not an importable package
Bootstrap failed: exit status 1

A file called main_easyjson.go with the following content is created

package  main

import (
  "github.com/mailru/easyjson/jwriter"
  "github.com/mailru/easyjson/jlexer"
)
func (* Foo ) MarshalJSON() ([]byte, error) { return nil, nil }
func (* Foo ) UnmarshalJSON([]byte) error { return nil }
func (* Foo ) MarshalEasyJSON(w *jwriter.Writer) {}
func (* Foo ) UnmarshalEasyJSON(l *jlexer.Lexer) {}

• Also, the output file is not gofmt'ed.

Significant regression in code generation performance.

Something changed today ( was good on friday ) that caused a significant performance degradation in the time it takes to generate code.

I'm still using 9d6630dc8c577b56cb9687a9cf9e8578aca7298a of easyjson.

Some form of system dependency changed in the last couple of days and I have no idea what it is.

Code generation for easyjson now takes 1 minute instead of 10 seconds.

Embedded reference struct should not need to be "new"'ed unless there are corresponding fields

Consider the following JSON:

{
  "Direct1":1,
  "Embedded2": 2,
  "Embedded3": 3
}

And given the following structs:

type A struct {
  Direct1 int
  *B
}

type B struct {
  Embedded2 int,
  Embedded3 int
}

When one generate the custom unmarshaling, there will be a line in the implementation that would read something like:

out.B = new(B)

regardless of if he has those embedded fields in the raw JSON string; e.g. when the JSON looks like:

{"Direct1":1}

he will still end up creating a new B. I think this B type shouldn't be created in this case because 1) it would waste additional memory, 2) it would make marshaling back into a different object when the string was directly marshaled from an A without B.

Embedded structs

Hi,

I have embedded structs and I get this problem "eaysjson_decode_... redeclared in this block ... previous declaration at ..."
I've noticed that structs are also copied where they were embedded. Do you have any options to turn this off?
Btw.: I tried to run easyjson file by file, and for the whole directory too, but I've got the same problem.

Nothing is generated for maps not in structs

Hi

I tried to generate un-/marshalers for the following code:

package gen

type Item struct {
    Val int
}

type TStruct struct {
    Mp map[string] Item
}

type TMap map[string] Item

The code was generated for TStruct and Item only, but map TMap was ignored. I modified genDecoder and genEncoder a the way like:

func (g *Generator) genDecoder(t reflect.Type) error {
    switch t.Kind() {
    case reflect.Map:
        return g.genMapDecoder(t)
    case reflect.Slice, reflect.Array:
        return g.genSliceArrayDecoder(t)
    default:
        return g.genStructDecoder(t)
    }
}

where genMapDecoder is similar to genSliceArrayDecoder except the Kind check. And it works!

So, I wonder why the current version doesn't allow to generate code for maps which not consisted in any struct?

Incompatible marshal'ing of net.IPNet

encoding/json marshal's net.IPNet as {"IP": "127.0.0.1", "Mask": ... }; it looks like easyjson expects the IP to be []byte's.

Null RawMessage field should be unmarshaled into []byte("null")

I mean this case:

https://play.golang.org/p/FzgG0R_O4s

Configurable Marshalling

Hi, not sure if you would think it's useful/good and would want a PR, but I added a feature to easy-json to allow some manipulation of the Marshaller at runtime.

Use case is that it's not uncommon to have a large struct and sometimes wanting to (dynamically) exclude some fields from marshaling (to send json subset of struct to endpoints)

See here,
tcolar#1

Dynamically marshalling specific fields

Basically I am also looking for pquerna/ffjson#150 (in some json library).

Wondering if you had ideas as to if/how you'd like to see support for it. I was thinking about adding an unexported function field to structs that need this support, and adding a conditional call to that function to the generated code.

type Selectable struct {
    A string
    B string
    easyjsonSelectField func(string) bool
}

s := Selectable{}
s.easyjsonSelectField = func(field string) bool {
    return true
}

This could be in addition to, or overriding omitempty.
I don't like that this requires modifying the struct, but I couldn't easily think of a better way to provide support to dynamically manage such a thing from generated code.

Happy to hear if you have any opinions on this functionality in general, or implementation.

Cannot use X (type Y) as type Z errors when compiling code with structs having custom types

Sorry for confusing title, but the problem is follows. I have a struct that looks like this:

type Message struct {
    UID       MessageID        `json:"uid"`
    Timestamp string           `json:"timestamp"`
    Info      *ClientInfo      `json:"info,omitempty"`
    Channel   Channel          `json:"channel"`
    Data      *json.RawMessage `json:"data"`
    Client    ConnID           `json:"client,omitempty"`
}

After generating *_easyjson.go file I tried to build my source code but got errors:

./message_easyjson.go:23: cannot use in.String() (type string) as type MessageID in assignment
./message_easyjson.go:35: cannot use in.String() (type string) as type Channel in assignment
./message_easyjson.go:47: cannot use in.String() (type string) as type ConnID in assignment
./message_easyjson.go:62: cannot use in.UID (type MessageID) as type string in argument to out.String
./message_easyjson.go:80: cannot use in.Channel (type Channel) as type string in argument to out.String
./message_easyjson.go:93: cannot use in.Client (type ConnID) as type string in argument to out.String
./message_easyjson.go:125: cannot use in.String() (type string) as type UserID in assignment
./message_easyjson.go:127: cannot use in.String() (type string) as type ConnID in assignment
./message_easyjson.go:162: cannot use in.User (type UserID) as type string in argument to out.String
./message_easyjson.go:166: cannot use in.Client (type ConnID) as type string in argument to out.String

Generated code - https://gist.github.com/FZambia/080a25f58ce81c510533

In my case types defined in this way:

type (
    Channel string
    ChannelID string
    UserID string
    ConnID string
    MessageID string
)

And ClientInfo is

type ClientInfo struct {
    User        UserID           `json:"user"`
    Client      ConnID           `json:"client"`
    DefaultInfo *json.RawMessage `json:"default_info,omitempty"`
    ChannelInfo *json.RawMessage `json:"channel_info,omitempty"`
}

How this can be fixed?

(Un)Marshal string to []byte

Is there any possibilities to unmarshal json-string to []byte and vice versa? Am I right that currently no?
It can be useful to reduce memory allocations and avoid unnecessary sting <-> []byte converting in application.

Anyway, I have a dummy fix for that and can formalize it in PR.

But the question is in what way to tell parser to process []byte that way. With cli falg e.g. -stringtobytes or better with tag e.g. easyjson:"string" for desired fields:

type T struct {
    name []byte `easyjson:"string"`
    ...
}

What do you think?

Lexer rewrites invalid token error on unmarshall

Unmarshall does not return correct error on invalid token type.
This is caused by call to lexer.Consumed which overwrites previous error.

Example struct:

type Model struct {
	A int    `json:"a"`
	B string `json:"b"`
}

Input:
{"a": "1", "b": "text"}

Got error:
parse error: invalid character ',' after top-level value near offset 9 of ', "b": "text"}'

Expected:
parse error: expected number near offset 9 of '1'

Stable version branches

We would like to be able to refer to a stable branch in order to avoid pulling in the current state of master for builds.

Lexer emits EOF on nil slices

Documentation states that all parsing errors should be LexerErrors, but it's possible for this not to be the case. This resulted in an error incorrectly being flagged as an internal error, when actually the client had simply omitted a field to be parsed.

EasyJSON disregards invalid input if a string prefix is valid

Discovered this quickly when throwing go-fuzz to detect behavioural differences between encoding/json and EasyJSON. EasyJSON seems to accept and stop parting if a prefix of a string is valid input. For example, all of the below lines are seen to be "valid" JSON strings:

null 4
{}{
null]

encoding/json returns an error when trying to parse these, EasyJSON accepts them

Give an explicit error message for nested maps with unsupported key type

When trying to apply easyjson to nested maps like map[string]map[uint32]string, it produces invalid code:

        for v2_name, v2_value := range in.Field {
            if !v2_first {
                out.RawByte(',')
            }
            v2_first = false
            out.String(string(v2_name))
            out.RawByte(':')
        }

which gives this error:

v2_value declared and not used

It would be nice to have an explicit error message, like we get with non-nested maps:

map type uint32 not supported: only string keys are allowed

encoding/json accepts case-insensitive keys, while easyjson doesn't

t2.go:

package t2

type S struct {
    A int
}

t2_test.go:

package t2

import (
    "encoding/json"
    "strings"
    "testing"
)

func TestCaseInsensitiveDecoding(t *testing.T) {
    d := json.NewDecoder(strings.NewReader(`{"a":1}`))
    var s S
    err := d.Decode(&s)
    if err != nil {
        t.Fatal(err)
    }
    if s.A != 1 {
        t.Fatal("expected 1, got", s.A)
    }
}

Without easyjson-generated code the test is passed; with it it fails:

t2_test.go:17: expected 1, got 0

This behaviour should be either documented or fixed.

easyjson doesn't respect TextUnmarshaler

https://golang.org/src/encoding/encoding.go?s=1851:1919#L36

For example, the following code will not work as expected with easyjson:

package main

import (
    "encoding/json"
    "fmt"
)

type Number int64

func (n *Number) UnmarshalText(text []byte) error {
    switch string(text) {
    case "ONE":
        *n = 1
    case "TWO":
        *n = 2
    default:
        *n = -1
    }
    return nil
}

type S struct {
    Index Number
}

func main() {
    var s S
    json.Unmarshal([]byte(`{"index": "ONE"}`), &s)
    fmt.Println(s)
}

jlexer: Lexer.Delim allocate two memory

Hi, Thank you for creating easyjson, it's so fast.
But when I profiling the generated code, I found Lexer.Delim() call r.errInvalidToken convert byte to slice of byte and then covert it to string.

I just modify Lexer, add errInvalidToken2(expected byte) just be used for Delim.

Invalid cross-device link

I'm not sure if I'm doing something wrong.

Here's the model file:

package models

// User is a person with an account. We only need to know their uid,
// email and location. Location can be USA or NON-USA
//easyjson:json
type User struct {
    UID      string `json:"uid" gorethink:"id"`
    Email    string `json:"email" gorethink:"email"`
    Location string `json:"location" gorethink:"location"`
    Services map[string]struct {
        Enabled bool   `json:"enabled,omitempty" gorethink:"enabled"`
        Status  string `json:"status,omitempty" gorethink:"status"`
    } `json:"services,omitempty" gorethink:"services,omitempty"`
}

Here's the output of the command:

msuppo@faster:~/Work/golang/src/github.com/matteosuppo/ctc-users$ easyjson -all api/models/user.go 
Bootstrap failed: rename /tmp/easyjson-out787644588 api/models/user_easyjson.go: invalid cross-device link

And here's the file generated

package  models

import (
  "github.com/mailru/easyjson/jwriter"
  "github.com/mailru/easyjson/jlexer"
)
func (* User ) MarshalJSON() ([]byte, error) { return nil, nil }
func (* User ) UnmarshalJSON([]byte) error { return nil }
func (* User ) MarshalEasyJSON(w *jwriter.Writer) {}
func (* User ) UnmarshalEasyJSON(l *jlexer.Lexer) {}

support non empty interface

easyjson doesn't support non empty interface members during marshal/unmarshal operations where as standard Go encoding/json does.

Required/Optional fields

For example we have such struct:

type SomeStruct struct {
    Key1 string
    Key2 string
    Key3 *string

where key3 is nullable, because zero-value is valid value for it.
And all these examples can be successfully unmarshalled:

{
    "key1": "value1",
    "key2": "value2",
    "key3": "value3"
}
{
    "key1": "value1",
    "key2": "value2",
        "key3": ""
}
{
    "key1": "value1",
    "key2": "value2",
}

But if key1 or key2 is missed then unmarshall must return an error.

What do you think about such mode?

-required by default flag

The same as for omit_empty flag.
I'll make PR if you'll merge it. (you need this feature)
Also added !required tag field
Just answer there.

How to parse top level array?

How to parse [{"a": "1"}] with easyjson ?

Format the output with gofmt.

As mentioned in #1, not formatting the output does not behave well with gofmt checks.

Fails to generate code for maps within structs

Attempting to run easyjson on this file generates the following errors:

types.go

package types

type UUID string
type Meter string

type Thing struct {
  UUID   UUID
  Length Meter
}

type Response struct {
  Things map[UUID]Thing
}

errors:

# ezjson_test
./types_easyjson.go:27: undefined: types in types.UUID
./types_easyjson.go:27: undefined: types in types.Thing
./types_easyjson.go:36: cannot use key (type string) as type UUID in map index
./types_easyjson.go:63: cannot use v2_name (type UUID) as type string in argument to out.String

time.Time does not marshall correctly

Marshalling and unmarshalling a struct like this leaves MyTime set to nil

type T struct {
MyTime time.Time json:"my_time"
}

encoding/json supports this

Unsafe in lexer.go

Hi all,

I'm working on an appengine application and one of my dependencies recently added an import to easyjson/jlexer. The problem is that https://github.com/mailru/easyjson/blob/master/jlexer/lexer.go#L207 is importing unsafe and deploying to appengine fails for code with unsafe.

Now, I understand that this is done to prevent copying the string bytes, but is there any way to rewrite it without unsafe? It would help all of us who write appengine apps.

Thanks,
tk

Parse error when parsing a degenerated case of JSON

t1.go:

package t1

type S struct{}

t1_test.go:

package t1

import (
    "encoding/json"
    "strings"
    "testing"
)

func TestDecodingNull(t *testing.T) {
    d := json.NewDecoder(strings.NewReader("null"))
    err := d.Decode(&S{})
    if err != nil {
        t.Fatal(err)
    }
}

Without easyjson-generated code the test is passed; with it it fails:

t1_test.go:13: parse error: expected {, got '' at byte 0.

null is a valid JSON.

Marshal methods are not generated for type aliases

Currently, it appears that type aliases are ignored when generating the marshaler methods. For instance,
on:

type Test struct {
    A string
}

type TestAlias Test

the file generated by easyjson -all does not contain any methods for TestAlias at all. This means that anything of type TestAlias is not an easyjson.Marshaler without an explicit cast.

Support for unexported structs

I continue my experiments with easyjson and results look really impressive at moment.

The problem I came across while adopting easyjson to Centrifugo is that it does not support code generation for unexported structs - i.e. I get lots of errors like cannot refer to unexported name libcentrifugo.apiCommand when run easyjson -all commands.go for this file. I can make all structs exported in the end of course but hope there is a simple way to support them too by easyjson.

Unrecognized package name error

Given a foo package containing at least two files and a map[string][]stuct member:

bar.go :

package foo

//easyjson:json
type Bar struct {
    Name  string
    Bozes map[string][]Boz
}

and boz.go :

package foo

//easyjson:json
type Boz struct {
    Name string
}

Then:

~/go/src/github.com/jeromenerf/foo % easyjson bar.go boz.go
# github.com/jeromenerf/foo
./bar_easyjson.go:36: undefined: foo in foo.Boz
./bar_easyjson.go:43: undefined: foo in foo.Boz
Bootstrap failed: exit status 2

(Running easyjson bar.go boz.go doesn't produce any error however).

The files produced :

// AUTOGENERATED FILE: easyjson marshaller/unmarshallers.

package foo

import (
    json "encoding/json"
    jlexer "github.com/mailru/easyjson/jlexer"
    jwriter "github.com/mailru/easyjson/jwriter"
)

var _ = json.RawMessage{} // suppress unused package warning

func easyjson_4347b5c1_decode_github_com_jeromenerf_foo_Bar(in *jlexer.Lexer, out *Bar) {
    if in.IsNull() {
        in.Skip()
        return
    }
    in.Delim('{')
    for !in.IsDelim('}') {
        key := in.UnsafeString()
        in.WantColon()
        if in.IsNull() {
            in.Skip()
            in.WantComma()
            continue
        }
        switch key {
        case "Name":
            out.Name = string(in.String())
        case "Bozes":
            if in.IsNull() {
                in.Skip()
            } else {
                in.Delim('{')
                if !in.IsDelim('}') {
                    out.Bozes = make(map[string][]foo.Boz)
                } else {
                    out.Bozes = nil
                }
                for !in.IsDelim('}') {
                    key := string(in.String())
                    in.WantColon()
                    var v1 []foo.Boz
                    in.Delim('[')
                    if !in.IsDelim(']') {
                        v1 = make([]Boz, 0, 4)
                    } else {
                        v1 = nil
                    }
                    for !in.IsDelim(']') {
                        var v2 Boz
                        (v2).UnmarshalEasyJSON(in)
                        v1 = append(v1, v2)
                        in.WantComma()
                    }
                    in.Delim(']')
                    (out.Bozes)[key] = v1
                    in.WantComma()
                }
                in.Delim('}')
            }
        default:
            in.SkipRecursive()
        }
        in.WantComma()
    }
    in.Delim('}')
}
func easyjson_4347b5c1_encode_github_com_jeromenerf_foo_Bar(out *jwriter.Writer, in Bar) {
    out.RawByte('{')
    first := true
    _ = first
    if !first {
        out.RawByte(',')
    }
    first = false
    out.RawString("\"Name\":")
    out.String(string(in.Name))
    if !first {
        out.RawByte(',')
    }
    first = false
    out.RawString("\"Bozes\":")
    if in.Bozes == nil {
        out.RawString(`null`)
    } else {
        out.RawByte('{')
        v3_first := true
        for v3_name, v3_value := range in.Bozes {
            if !v3_first {
                out.RawByte(',')
            }
            v3_first = false
            out.String(string(v3_name))
            out.RawByte(':')
            out.RawByte('[')
            for v4, v5 := range v3_value {
                if v4 > 0 {
                    out.RawByte(',')
                }
                (v5).MarshalEasyJSON(out)
            }
            out.RawByte(']')
        }
        out.RawByte('}')
    }
    out.RawByte('}')
}
func (v Bar) MarshalJSON() ([]byte, error) {
    w := jwriter.Writer{}
    easyjson_4347b5c1_encode_github_com_jeromenerf_foo_Bar(&w, v)
    return w.Buffer.BuildBytes(), w.Error
}
func (v Bar) MarshalEasyJSON(w *jwriter.Writer) {
    easyjson_4347b5c1_encode_github_com_jeromenerf_foo_Bar(w, v)
}
func (v *Bar) UnmarshalJSON(data []byte) error {
    r := jlexer.Lexer{Data: data}
    easyjson_4347b5c1_decode_github_com_jeromenerf_foo_Bar(&r, v)
    return r.Error()
}
func (v *Bar) UnmarshalEasyJSON(l *jlexer.Lexer) {
    easyjson_4347b5c1_decode_github_com_jeromenerf_foo_Bar(l, v)
}

and

// TEMPORARY AUTOGENERATED FILE: easyjson stub code to make the package
// compilable during generation.

package  foo

import (
  "github.com/mailru/easyjson/jwriter"
  "github.com/mailru/easyjson/jlexer"
)

func (* Boz ) MarshalJSON() ([]byte, error) { return nil, nil }
func (* Boz ) UnmarshalJSON([]byte) error { return nil }
func (* Boz ) MarshalEasyJSON(w *jwriter.Writer) {}
func (* Boz ) UnmarshalEasyJSON(l *jlexer.Lexer) {}

type EasyJSON_exporter_Boz *Boz

Fails to property generate if import has a .

if your import has a dependency with a "." in it such as
import "gopkg.in/inf.v0"
This will fail to generate the easyjson file properly. It will treat the package as inf.v0 instead of as inf

Unmarshalling not as fast as expected?

Comparing easyjson to encoding/json, I am not seeing much speed improvements.

I am using easyjson.UnmarshalFromReader to Unmarshall ~2000 objects per second and am noticing an average of 380ms per 2k objects. Using the same test on the same data, I am seeing an average of 420ms per 2k objects with encoding/json.

Are there some optimisation possibilities that I am missing or is this expected behavior?

Inline struct fields are omitted in case of name collision

Consider following code:

type Outer struct {
    Inner
    A int64       `json:"a"`
    B interface{} `json:"b"`
}

type Inner struct {
    A map[string]float64 `json:"innerA"`
    B string             `json:"innerB"`
}

Serializing Outer{} using encoding/json gives us {"innerA":null,"innerB":"","a":0,"b":null} (snippet), but if we try to do the same in easyjson fields of inner struct are supressed by fields of the outer struct.

Unmarshal to array of struct

Hi,

Given a struct Test, that has marshal and unmarshal interfaces generated, is it possible to unmarshal data to []Test?

I have tried various alternatives like:

var ArrTestType []Test

type ArrTest struct {
    ArrTestType //anonymous embedded field
}

but it doesnt work. In this case, easyjson complains that it expects a struct (for the embedded field).

go get fails on x32 systems

go get -u github.com/mailru/easyjson/...
# github.com/mailru/easyjson/benchmark
/opt/go/gopath/src/github.com/mailru/easyjson/benchmark/data_var.go:7: constant 250126199840518145 overflows int
/opt/go/gopath/src/github.com/mailru/easyjson/benchmark/data_var.go:12: constant 24012619984051000 overflows int
# github.com/mailru/easyjson/tests
/opt/go/gopath/src/github.com/mailru/easyjson/tests/data.go:91: constant 4294967295 overflows int
/opt/go/gopath/src/github.com/mailru/easyjson/tests/data.go:94: constant 4294967295 overflows int
/opt/go/gopath/src/github.com/mailru/easyjson/tests/data.go:103: constant 4294967295 overflows int
/opt/go/gopath/src/github.com/mailru/easyjson/tests/data.go:106: constant 4294967295 overflows int
/opt/go/gopath/src/github.com/mailru/easyjson/tests/data.go:194: constant 4294967295 overflows int
/opt/go/gopath/src/github.com/mailru/easyjson/tests/data.go:197: constant 4294967295 overflows int

undefined: net in net.IP

Building after running easyjson on the struct below fails with an error undefined: net in net.IP (because net doesn't get added to imports).

type S struct {
    Time       time.Time
    Client struct {
        IP   net.IP
        Mask uint8
    }
}

just a suggestion: add a method called String()

thank you for selfless dedication, this project is great.

As title describe, usually we want to print some json infomation, plz think about it, thanks.

easyjson is not so vendor friendly

Consider the following code:

import "github.com/repo1/package2"

//easyjson:json
type Gen1 struct {
  A package2.Type1 `json:"a"`
}

Given that github.com/repo1/package2 is already vendored, e.g. there is a vendor/github.com/repo1/package2 directory along with the go source code above. Running the easyjson binary will create an import line that looks somelike:

import package2 "github.com/foo/bar/vendor/github.com/repo1/package2"

Instead, it should just be:

import package2 "github.com/repo1/package2"

My go env just FYI:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/stevejiang/developments/go"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.6.2/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.6.2/libexec/pkg/tool/darwin_amd64"
GO15VENDOREXPERIMENT="1"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fno-common"
CXX="clang++"
CGO_ENABLED="1"

file X is not in GOPATH error when GOPATH ends with `/`

Hello, here is a problem I came across when trying easyjson – when I run easyjson.go myfile.go I get an error from title.

After investigating a bit I found the reason: my GOPATH ends with / so it does not pass a check in this line.

The easiest fix I can imagine is sth like:

import "path/filepath"
...
prefix := filepath.Join(p, "src") + "/"

But maybe there is a cleaner way to do it.

Is this project still maintained?

I have 3 pull requests open that haven't even been looked at in months.

If you are not planning on maintaining this anymore, maybe add someone else who can maintain it?

fails to correctly encode non-pointer structures.

package main

import (
    "encoding/json"
    "fmt"
)

//go:generate easyjson -snake_case example.go

//easyjson:json
type obj struct {
    Field1 string
    Field2 string
}

func main() {
    var (
        buffer []byte
        err    error
    )
    details := obj{
        Field1: "foo",
        Field2: "bar",
    }

    if buffer, err = json.Marshal(details); err != nil {
        fmt.Println("marshal error", err)
        return
    }

    fmt.Println("bad", string(buffer))

    if buffer, err = json.Marshal(&details); err != nil {
        fmt.Println("marshal error", err)
        return
    }

    fmt.Println("good", string(buffer))
}

EasyJSON panics on invalid object data

Parsing a string containing a single quote, ", cause EasyJSON to panic:

panic: runtime error: slice bounds out of range

goroutine 1 [running]:
panic(0x1030e0, 0x420206100)
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/goroot/src/runtime/panic.go:500 +0x1a1
github.com/mailru/easyjson/jlexer.(*Lexer).Consumed(0x420231d78)
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/gopath/src/github.com/mailru/easyjson/jlexer/lexer.go:538 +0x3a4
github.com/connor4312/easyfuzz.easyjsonF902965fDecodeGithubComConnor4312Easyfuzz(0x420231d78, 0x420231e60)
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/gopath/src/github.com/connor4312/easyfuzz/fuzz_easyjson.go:115 +0x105b
github.com/connor4312/easyfuzz.(*Foo).UnmarshalJSON(0x420231e60, 0x842026d000, 0x1, 0x200000, 0x420231e90, 0x8)
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/gopath/src/github.com/connor4312/easyfuzz/fuzz_easyjson.go:233 +0xa6
github.com/connor4312/easyfuzz.Fuzz(0x842026d000, 0x1, 0x200000, 0x0)
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/gopath/src/github.com/connor4312/easyfuzz/fuzz.go:29 +0x9e
go-fuzz-dep.Main(0x12aed0)
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/goroot/src/go-fuzz-dep/main.go:49 +0xe8
main.main()
	/var/folders/nz/3m0w2xzd1hq3xnhdfq83pq_00000gn/T/go-fuzz-build368983030/gopath/src/github.com/connor4312/easyfuzz/go.fuzz.main/main.go:10 +0x2d
exit status 2

Test struct:

type Foo struct {
	A         int
	B         float32
	C         string
	PtrA      *int
	PtrB      *float32
	PtrC      *string
	MapStrInt map[string]int
	MapStrAny map[string]interface{}
}

Wrong behaviour for embedded type

Hi

type Parent struct {
  ParentMember string
  Child
}

type Child struct {
  ChildMember string
}

1. If Child has an easyjson implementation.
2. Then attempt to marshal Parent.
3. Then ParentMember is not present in output! (unexpected)

It's caused by parent inheriting child's implementation of json.Marhsaller interface.

I don't know if there's any possible solution. I hope there is. But maybe in README.MD should add a note saying "any type that embeds a type using easyjson must itself also use easyjson".