matttproud/golang_protobuf_extensions

Support for streaming Protocol Buffer messages for the Go language (golang).

Overview

Note: This Go package is on the road to retirement. You can learn more at this blog post: https://matttproud.com/blog/posts/retiring-pbutil.html.

This repository provides various Protocol Buffer feature extensions for the Go programming language (golang), namely support for record length-delimited message streaming.

Java	Go
MessageLite#parseDelimitedFrom	pbutil.ReadDelimited
MessageLite#writeDelimitedTo	pbutil.WriteDelimited

Because Code Review 9102043 is destined to never be merged into mainline (i.e., never be promoted to formal goprotobuf features), this repository will live here in the wild.

Documentation

We have generated Go Doc documentation here.

Testing

golang_protobuf_extensions's Issues

tests fail with Go 1.5 tip

mwhudson@glamdring:~$ go version
go version devel +0acecb7 Wed Jul 22 20:27:24 2015 +0000 linux/amd64
mwhudson@glamdring:~$ go test -v github.com/matttproud/golang_protobuf_extensions/pbutil
=== RUN   TestWriteDelimited
--- PASS: TestWriteDelimited (0.00s)
=== RUN   TestReadDelimited
--- PASS: TestReadDelimited (0.00s)
=== RUN   TestEndToEndValid
--- PASS: TestEndToEndValid (0.00s)
=== RUN   TestFuzz
--- FAIL: TestFuzz (0.00s)
panic: reflect: call of reflect.Value.NumField on zero Value [recovered]
    panic: reflect: call of reflect.Value.NumField on zero Value

goroutine 8 [running]:
testing.tRunner.func1(0xc8200a21b0)
    /opt/opensource/go/src/testing/testing.go:450 +0x171
reflect.flag.mustBe(0x0, 0x19)
    /opt/opensource/go/src/reflect/value.go:199 +0xa6
reflect.Value.NumField(0x0, 0x0, 0x0, 0x0)
    /opt/opensource/go/src/reflect/value.go:1146 +0x2a
github.com/matttproud/golang_protobuf_extensions/pbtest.sanitizeGenerated(0x7f92a0ca6ee8, 0x0, 0x1, 0x0, 0x0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbtest/quick.go:53 +0x2f8
github.com/matttproud/golang_protobuf_extensions/pbtest.sanitizeGenerated(0x7f92a0ca6e40, 0xc820033b00, 0x0, 0x0, 0x0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbtest/quick.go:64 +0x83e
github.com/matttproud/golang_protobuf_extensions/pbtest.SanitizeGenerated(0x7f92a0ca6e40, 0xc820033b00, 0x0, 0x0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbtest/quick.go:33 +0x40
github.com/matttproud/golang_protobuf_extensions/pbutil.rndMessage(0xc820011240, 0x0, 0x0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbutil/all_test.go:273 +0x3bd
github.com/matttproud/golang_protobuf_extensions/pbutil.rndMessages(0xc820011240, 0x0, 0x0, 0x0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbutil/all_test.go:284 +0xb7
github.com/matttproud/golang_protobuf_extensions/pbutil.TestFuzz.func1(0x0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbutil/all_test.go:292 +0x4a
reflect.Value.call(0x587e40, 0xc82000a9a0, 0x13, 0x60dca8, 0x4, 0x749988, 0x0, 0x0, 0x0, 0x0, ...)
    /opt/opensource/go/src/reflect/value.go:432 +0x120a
reflect.Value.Call(0x587e40, 0xc82000a9a0, 0x13, 0x749988, 0x0, 0x0, 0x0, 0x0, 0x0)
    /opt/opensource/go/src/reflect/value.go:300 +0xb1
testing/quick.Check(0x587e40, 0xc82000a9a0, 0x725e60, 0x0, 0x0)
    /opt/opensource/go/src/testing/quick/quick.go:270 +0x421
github.com/matttproud/golang_protobuf_extensions/pbutil.TestFuzz(0xc8200a21b0)
    /opt/opensource/gopath/src/github.com/matttproud/golang_protobuf_extensions/pbutil/all_test.go:317 +0x102
testing.tRunner(0xc8200a21b0, 0x723248)
    /opt/opensource/go/src/testing/testing.go:456 +0x98
created by testing.RunTests
    /opt/opensource/go/src/testing/testing.go:561 +0x86d

goroutine 1 [chan receive]:
testing.RunTests(0x66f140, 0x723200, 0x4, 0x4, 0x40ec01)
    /opt/opensource/go/src/testing/testing.go:562 +0x8ad
testing.(*M).Run(0xc820043ef8, 0x1b)
    /opt/opensource/go/src/testing/testing.go:494 +0x70
main.main()
    github.com/matttproud/golang_protobuf_extensions/pbutil/_test/_testmain.go:60 +0x116
exit status 2
FAIL    github.com/matttproud/golang_protobuf_extensions/pbutil 0.006s
mwhudson@glamdring:~$ 

I haven't checked but I'm 99% certain this is because testing/quick.Value can return a nil pointer now (see very bottom of http://tip.golang.org/doc/go1.5).

Resolve CVE-2021-3121

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3121

pbutil: dependencies/tests fail with github.com/golang/protobuf v1.1.0

github.com/golang/protobuf v1.1.0 (and later) have removed the testdata directory so it can no longer be imported. This breaks tests:

$ go test
# github.com/matttproud/golang_protobuf_extensions/pbutil
all_test.go:27:2: cannot find package "github.com/golang/protobuf/proto/testdata" in any of:
	/home/mark/go1.10.2/src/github.com/golang/protobuf/proto/testdata (from $GOROOT)
	/home/mark/src/github.com/golang/protobuf/proto/testdata (from $GOPATH)
FAIL	github.com/matttproud/golang_protobuf_extensions/pbutil [setup failed]

I noticed this when attempting to use vgo vendor with a module that depends on github.com/prometheus/client_golang/prometheus. vgo is unable to enumerate dependencies since pbutil's dependencies are broken.

Tag a v1.0.2 release

I noticed that the last release is missing #15, and does not yet have a go.mod;

v1.0.1...c182aff

Some projects started to pin the dependency to that commit, instead of the last release; would it be possible to tag a v1.0.2 from master? @matttproud

Required field "Coordinates.{Unknown}" not set

Dear @matttproud,

I hope you are well since #12 !

We are just experiencing issue with the unmarshal of the same flow and proto file mentioned in the issue above. Here is the issue raised by the pbutil.ReadDelimited function: proto: required field "Coordinates.{Unknown}" not set

Something weird is that, according to the data provider, nothing has changed on the proto file. Even weirder, some message are marshalable and some aren't.

My guess is an issue on the emitter side (since they are updating their stuffs and we didn't touch the receiver code since 1 year) but it's hard to know what's going on.

Maybe you have some clues on what's happening as you might have experienced the same kind of errors while developing this lib ? I'd say that the way they build their frames is not consistent and subject to EOF prematurely before being fully constructed.

Thanks a lot for your work on this lib and it would be amazing if you have any idea of what's going on.

Confidence level

Dear Matt,

Really enjoying the great work on this lib.

Do you think it's safe to use this in production and if not, what do you think needs work ? I'm happy to help you on that.

We'll need to decode incoming protobuf from an external Java application advising the use of their protobuf like this :

// parse all frames
CoordinatesFrame frame = CoordinatesFrame.parseDelimitedFrom(in);
while (frame != null) {
	System.out.println("FRAME: " + frame.getTimestamp() + " " + frame.getCoordinatesCount());
	frame = CoordinatesFrame.parseDelimitedFrom(in);
}

So far using you extensions with the generated .pb.go is working. I'm using it like this :

inBuffer := bytes.NewReader(in)

i := 0
n := 1
for n > 0 {
	coordinates := &CoordinatesFrame{}
	n, err = pbutil.ReadDelimited(inBuffer, coordinates4)
	if err != nil {
		log.Errorf("Kaput unmarshaling proto: %s", err)
	}

	// Use coordinates here

	i++
}

Regards from Belgium,
Albin.

protodelim is now included in official Go implementation

I just notice it has been implemented in
https://pkg.go.dev/google.golang.org/[email protected]/encoding/protodelim

Would it make sense to suggest people to migrate to the official one once it has been tagged/released?

Breaking change

After the release of v1.0.3 this library is no longer backwards compatible with the old proto.Message interface

V1 Message Interface

type MessageV1 interface {
	Reset()
	String() [string](https://pkg.go.dev/builtin#string)
	ProtoMessage()
}

After the import of new protobuf package from google.golang.org/protobuf/proto proto.Message is https://pkg.go.dev/google.golang.org/[email protected]/reflect/protoreflect#ProtoMessage

type ProtoMessage interface{ 
   ProtoReflect() Message 
}

This currently breaks https://github.com/prometheus/common/blob/main/expfmt/decode.go

go/pkg/mod/github.com/prometheus/[email protected]/expfmt/decode.go:89:38: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.ReadDelimited:
        *io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)

go/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:120:40: cannot use v (variable of type *io_prometheus_client.MetricFamily) as type protoreflect.ProtoMessage in argument to pbutil.WriteDelimited:
        *io_prometheus_client.MetricFamily does not implement protoreflect.ProtoMessage (missing ProtoReflect method)

Could you please remove this release and tag it with v2.0.0 or something?