shamaton/msgpack

easier, faster, but extendable MessagePack Serializer for Golang. / msgpack.org[Go]

MessagePack for Golang

📣 Notice

If your application serializes only primitive types, array, map and struct, code generation is also recommended. You can get the fastest performance with msgpackgen.

Features

• Supported types : primitive / array / slice / struct / map / interface{} and time.Time
• Renaming fields via msgpack:"field_name"
• Omitting fields via msgpack:"-"
• Supports extend encoder / decoder
• Can also Encoding / Decoding struct as array

Installation

Current version is msgpack/v2.

go get -u github.com/shamaton/msgpack/v2

Quick Start

package main

import (
  "github.com/shamaton/msgpack/v2"
  "net/http"
)

type Struct struct {
	String string
}

// simple
func main() {
	v := Struct{String: "msgpack"}

	d, err := msgpack.Marshal(v)
	if err != nil {
		panic(err)
	}
	r := Struct{}
	if err =  msgpack.Unmarshal(d, &r); err != nil {
		panic(err)
	}
}

// streaming
func handle(w http.ResponseWriter, r *http.Request) {
	var body Struct
	if err := msgpack.UnmarshalRead(r, &body); err != nil {
		panic(err)
    }
	if err := msgpack.MarshalWrite(w, body); err != nil {
		panic(err)
    }
}

Benchmark

This result made from shamaton/msgpack_bench

License

This library is under the MIT License.

msgpack's Issues

msgpack.Marshal is panicking on "index out of range"

I don't currently have a way to reproduce this issue, but I am seeing this panic sometimes:

runtime error: index out of range [415989989] with length 415989989
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/set.go", line 66, in (*encoder).setByte1Int
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/bool.go", line 11, in (*encoder).writeBool
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/map.go", line 307, in (*encoder).writeFixedMap
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/encoding.go", line 370, in (*encoder).create
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/struct.go", line 227, in (*encoder).writeStructMap
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/struct.go", line 182, in (*encoder).writeStruct
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/encoding.go", line 382, in (*encoder).create
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/encoding.go", line 389, in (*encoder).create
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/encoding.go", line 378, in (*encoder).create
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/struct.go", line 227, in (*encoder).writeStructMap
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/struct.go", line 182, in (*encoder).writeStruct
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/encoding.go", line 382, in (*encoder).create
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/internal/encoding/encoding.go", line 46, in Encode
  File "/go/pkg/mod/github.com/shamaton/msgpack/[email protected]/msgpack.go", line 18, in Marshal

It seems like something in the msgpack code is incorrectly calculating the final size of the byte slice to hold the encoded version of the object. The object being encoded looks something like this:

type A struct {
	Field1 B
	Field2 map[string]B
}

type B struct {
	Field1 *string
	Field2 time.Time
	Field3 map[string]bool
	Field4 []byte
}

When marshalling A, it sometimes panics in the writeBool method, which I assume is happening as it encodes B.Field3 for some value of B. I'm not sure there's any significant to the place that the panic happens, since there was clearly a miscalculation earlier that tries to create the correct size of byte slice, and the fact that it happens here is likely just a coincidence.

Obviously, the index out of range error indicates this is going to be a really large messagepack object (415 megabytes, at least), and that's something I'm hoping to make smaller in my application code... I just wouldn't expect it to cause an out of range error. I've also seen it happen with objects as small as 227 megabytes.

I don't know if you have any ideas of how this could be happening. If I had found a way to reliably reproduce this error, I would be trying to debug it myself, but it's hard when this error doesn't happen on my local machine.

Support for Protocol Buffers

Any plans for adding support for un/marshalling PB messages? Specifically the generated tags and oneof fields.

Build error: math.MaxUint32 overflows int

Hi!
I've got this error when building the lib with go 1.12.1 windows/386 on Windows 10

C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\byte.go:22:14: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\byte.go:36:14: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\encoding.go:110:15: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\encoding.go:158:15: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\encoding.go:195:15: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\map.go:267:14: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\slice.go:107:14: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\struct.go:94:14: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\struct.go:141:14: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\struct.go:197:16: constant 4294967295 overflows int
C:\Users\_\go\pkg\mod\github.com\shamaton\[email protected]\internal\encoding\struct.go:197:16: too many errors

Way to serialize int64 (unix timestamp) into timestamps via explicit struct field param

// User struct describing a user account
type User struct {
	Key         string   `json:"_key,omitempty"`
	Email       string   `json:"email"`
	EmailMD5    string   `json:"emailmd5"`
	Username    string   `json:"username"`
	Description string   `json:"description,omitempty"`
	Verifier    string   `json:"verifier,omitempty"`
	Created     int64    `json:"created,omitempty" msgpack:"created,timestamp"` // <- this
	Logins      []int64  `json:"logins,omitempty" msgpack:"logins,timestamp"`
	Sessions    []int64  `json:"sessions,omitempty" msgpack:"sessions,timestamp"`
	Roles       []Role   `json:"roles,omitempty"`
	Friends     []string `json:"friends,omitempty"`
	Exp         int64    `json:"exp,omitempty"`
	Subscriber  bool     `json:"subscriber,omitempty"`
}

Also is there a way to just default to json values when encoding? You know just use the json names.
Sorry if that's already the case.

Anyway. So far, this looks like a pretty good library and I'm keen on using it if it would be able to serialize unix epochs into proper msgpack timestamps with an extra flag or something. 😄

Why were the patch versions for CVE-2022-41719 released so late?

Hello, we are a research team working on Golang. During our investigation, we found CVE-2022-41719 was addressed in commit 2cb19ad. However, we noticed that the patch version (v2.1.1) was released after long time (40 days). We are curious about the reasons behind the delayed release of the patch version, as it may hinder the efficient distribution of patches to downstream users. Could the reason be

1.Issues with testing and CI checking.

2.Other commits have to be incorporated into one release.

3.By convention, versions are not frequently released.

4.Other reasons.

Thank you for your attention, and we look forward to receiving your reply.

Many panics/crashes when fuzzing

Hi there,

I've been fuzzing this library using the excellent go-fuzz fuzzer. It has produced quite a few panics in the Unmarshal functionality. These panics can have security implications and can lead to, for example, denial-of-service. Ideally this library would be resilient to potentially malicious msgpack payloads.

I used the following code to test Unmarshal against fuzzed inputs:

var r interface{}
                                  
err = msgpack.Unmarshal(data, &r)
if err != nil {
	panic(err)
}

This produced a number of crashes. I see three general types of crashes:

1:panic: runtime error: slice bounds out of range [:6] with capacity 1
1:panic: runtime error: index out of range [23] with length 23
1:panic: runtime error: hash of unhashable type map[interface {}]interface {}

There are many more like this, but they're the same crash with different values.

I've attached a zipfile with files that contain msgpack data that produce the panics. Calling Unmarshal on the data like the above code snippet should reproduce the crashes. Due to potential security implications here, it would be beneficial if this msgpack implementation was resilient to these payloads, and produced errors on invalid input instead of panics. Is it possible to fix the code such that it is?

Thank you for creating this library, and let me know if you have any questions!

crashers.zip

How to extend the library to new types?

I have structs with fields a type msgpack does not understand. Fortunately the type has a simple way to be built and stored as sequence of bytes, so I expected to be fairly straightforward to extend the library to support it.

However, when I try to use it I get the error "msgpack : invalid code c7 decoding struct". 0xc7 is of course the code of Ext8 that I used since we are speaking of 40-70 bytes per element.

I have the feeling I am missing something obvious...

Here is the relevant code, minimized.

type MyEncoder struct {}

type MyDecoder struct {}

func (*MyEncoder) Code() int8 {
  return -3
}

func (*MyEncoder) Type() reflect.Type {
  return reflect.TypeOf(MyThingy)
}

func (*MyEncoder) CalcByteSize(value reflect.Value) (int, error) {
  m := value.Interface().(MyThingy)
  return 2 + len(m.ToBytes()), nil
}

func (c *MyEncoder) WriteToBytes(value reflect.Value, offset int, bytes *[]byte) int {
  m := []byte(value.Interface().(MyThingy).ToBytes())

  (*bytes)[offset] = msgpackDef.Ext8
  offset++

  (*bytes)[offset] = byte(c.Code())
  offset++

  (*bytes)[offset] = byte(len(m))
  offset++

  for _, c := range m {
  	(*bytes)[offset] = c
  	offset++
  }

  return offset
}

func (*MyDecoder) Code() int8 {
  return -3
}

func (c *MyDecoder) IsType(offset int, d *[]byte) bool {
  return (*d)[offset] == msgpackDef.Ext8 && (*d)[offset+1] == byte(c.Code())
}

func (*MyDecoder) AsValue(offset int, k reflect.Kind, d *[]byte) (interface{}, int, error) {
  nrBytes := int((*d)[offset+2])
  m := MyThingy{}
  mm, err := m.FromBytes((*d)[3 : 3+nrBytes])
  return mm, 3 + nrBytes, err
}

Panic on zero-length input

It seems like msgpack.Unmarshal(data, &value) panics if data is zero-length.

I would just expect to get an error back that the input isn't long enough, not a panic. I'm not sure if this was an intentional design choice or not, but for comparison, json.Unmarshal just returns an error in this case, not a panic.

I'm adding a check in my code to avoid calling msgpack.Unmarshal with a zero-length value now, so I can work around it in the mean time.

Unmarshalled Strings Unsafely Share Byte Slices

If the bytes in the byte slice passed to msgpack.Unmarshal change after msgpack.Unmarshal returns, any of the unmarshaled strings may change their contents, violating the expectation that Go strings are immutable. This is probably due to the use of unsafe.Pointer in decoder.asString. This is surprising behavior which can really screw things up in Go if the original byte slice was a buffer whose contents are replaced on each message.

See https://play.golang.org/p/YQyb8ie8Qqd for a simple repro of the issue.

The simplest fix I can see is to document this behavior and warn not to reuse the underlying byte array until any strings are expunged. Discontinuing this behavior would likely have a dramatic negative impact on performance. The most complex fix would be to make the behavior of asString configurable, which would let the users decide if they want to sacrifice sanity for efficiency. (And this is an insane behavior -- consider what happens if the resulting string was used as a map key by an unsuspecting user.)

Structs with map[string] to nested structs showing only zero values after marshal and unmarshal

Hi @shamaton I have been enjoying the simplicity and functionality of this library and it has been a great fit for my projects.
I recently came across an issue which is possibly just user error (I was hoping this is allowed) but I wanted to reach out in case its a bug.

I have a large struct which contains many maps and slices, some of which are maps with value types that are nested structs. Those structs only contain primitive types, meaning everything can theoretically be decomposed into primitive types and serialized, however I am seeing misbehavior on these nested structs. When I print the struct instances before marshalling, everything looks perfectly correct. Then I am using the msgpack.Marshal function to convert the struct instance into a slice of bytes which I am then writing to a file using a zstd Writer. When I uncompress the file, unmarshal the data using msgpack.Unmarshal, and try to print the data structure I see all the keys deserializing correctly, but the values which are nested structs show all of their contents set to 0. Currently I am not suspecting the compression library to be the issue since I have tried many other compression algos.

I'll share some pseudo code here so you can get the idea, and maybe you have some advice for me.

type foo struct {
	size  int
	start float64
	stop  float64
	step  float64
}

type bar struct {
	num   int
	b0    []int
	b1    []int
	b2    []int
	low   []int
	high  []int
	name  string
	group string
}

type baz struct {
	Count  int
	Foos   map[string](foo)
	Bars   map[string](bar)
	Tests  []string
	Names  map[string](struct{})
	Sizes  map[string](int)
	Plugs  map[string]([]float32)
	Chunks map[string]([][]float32)
	Banks  map[string](map[string]([]byte))
	Stores map[string](map[string]([]byte))
}

func main {

	var myBaz baz

	// Populate a huge myBaz structure here ~10 GB

	fmt.Println(myBaz)

	d, err := msgpack.Marshal(myBaz)
	if err != nil {
		fmt.Println("Error trying to marshal archive file.")
		os.Exit(1)
	}
	fileName := "example.zst"
	f, err := os.Create(fileName)
	if err != nil {
		fmt.Println(err)
		os.Exit(1)
	}
	zstd.WithEncoderConcurrency(0)
	zstd.WithEncoderLevel(1)
	w, err := zstd.NewWriter(f)
	if err != nil {
		fmt.Println("Couldn't create zstd writer.")
		os.Exit(1)
	}
	w.Write(d)
	w.Close()
	fmt.Println("Wrote to", fileName)
	f.Close()

	// Come back in a different command line option to read in the file.

	f, err := os.Open(fileName)
	if err != nil {
		fmt.Println("Error opening archive file.")
		os.Exit(1)
	}
	zstd.WithDecoderConcurrency(0)
	r, err := zstd.NewReader(f)
	if err != nil {
		fmt.Println("Couldn't create zstd reader.")
		os.Exit(1)
	}
	var rawData []byte
	rawData, err = io.ReadAll(r)
	if err != nil {
		fmt.Println("Couldn't read all decompressed data into buffer.")
		os.Exit(1)
	}
	r.Close()
	var myNewBaz baz
	err = msgpack.Unmarshal(rawData, &myNewBaz)
	if err != nil {
		fmt.Println("Couldn't unmarshall data from archive file.")
		os.Exit(1)
	}
	f.Close()

	fmt.Println(myNewBaz)

}

In the example above after unmarshalling an instance of baz like "myNewBaz" I would be able to see all of the expected contents from every field exactly as they appeared before I marshalled the original struct except for "Foos" and "Bars" which again were the only fields which had a map of string to nested structs.

I'm curious if there is a way to address this without having to rewrite my code to use slices of strings or something to perform the purpose of the foo and bar structs above. I also thought about unrolling the structs into a pile of maps inside baz, but it feels dirty so I'm hoping there is a better way. (and I'm also hoping I don't need to do code generation since its new to me and I'm worried about complicating my code for others on my team)

Thanks in advance.