Comments (6)
This would be a welcome contribution because Vitess does use some well-known proto-types.
I'm happy to send a PR, but one thing I'd ask the library maintainers to chime in on is whether it's acceptable to have the generated code reference global helper functions from a package within this module
This would be something new, as the generated protobuf files don't have any dependencies, but at the same time it feels more elegant than throwing up the same (de)serialization code over and over for the same well-known types in different packages. Since this package doesn't really have any dependencies besides protobuf
itself, I say let's try it this way.
If you have any arguments why generating package-private functions would be better I'd love to hear them too. Cheers!
from vtprotobuf.
I was looking into this a bit and it seems fairly complex.
I think we either need to use our own copy of the wellknown types, which isn't great for compatibility (I think this is how gogo did it).
OR we have totally specially logic and do not implement the typical interfaces
WDYT?
Since we cannot just add a method on a foreign type unfortunately
from vtprotobuf.
I did some experimentation on this (marshaling https://github.com/envoyproxy/envoy/blob/5fefd9d4f78d169e3cb71f4e2c53a1b5d575334b/api/envoy/config/endpoint/v3/endpoint_components.proto#L88, fwiw). We have a variety of benchmarks for different sizes of protobufs.
Not all numbers are the end-to-end benchmark numbers, they do a bit more than just protobuf encoding - but its 75% proto.
With just plain vtprotobuf, we only saw a 5-10% improvement.
With wellknown types handled, we say a 35-65% improvement in CPU, and up to 80% allocation reduction(!). It actually is enough to bring protobuf time to under 50% of our costs (from 75%).
Note this is just a hacky prototype. I have it at https://github.com/howardjohn/vtprotobuf/pull/new/gen/wellknown for reference. It current generates the _vtproto.pb.go
, but then they are copied into a fork of google.golang.org/protobuf
to work around issues in #54 (comment)
from vtprotobuf.
Interestingly I think the motivation for reflection in core is to reduce binary sizes (?). Builtin types are used almost everywhere so would have big impact to just have these in core (replacing Marshal I mean not the VT extension)
Could probably even have completely hand written logic for some of the trivial ones like wrapper.Uint32...
from vtprotobuf.
We also are not getting the full value proposition out of this due to parsing several million timestamppb.Timestamp
per second.
Happy to help if there's anything I can do on my end to get this through
from vtprotobuf.
I also noticed #80 which is another attempt to support WKT
from vtprotobuf.
Related Issues (20)
- Uncompilable code generated for repeated groups
- Duplicated code generated for GRPC client/server HOT 1
- Panic after WKT commit: assignment to entry in nil map HOT 2
- Extensions are not encoded
- docs/feature: Nested ReturnToVTPool() HOT 4
- bug: oneof field that includes bytes field yields invalid ResetVT method
- Supporting Embedded and Nullable features from gogo proto HOT 2
- bug: Broken ResetVT generation for optional message HOT 1
- bug: grpc client methods does not use qualified idents
- feature: poolable messages paths with wildcard HOT 2
- unmarshal_unsafe feature produced code does not compile: "Cannot use intStringLen (type int) as the type IntegerType" HOT 1
- Run vtproto multiple times on different files of the same package HOT 3
- bug: IsMap not checked on message field's ReturnToVTpool
- bug: UnmarshalUnsafe panics when map values are empty strings HOT 2
- Request for New Release that Includes Well-Known Type Support HOT 4
- gRPC codec has different semantic on Unmarshal than the default one HOT 1
- v0.6.0 is creating issues in ubuntu20 HOT 2
- Offering assistance and discussing upkeep / releases for vtprotobuf HOT 8
- wrong pool unmarshal slize HOT 3
- Add optional unsafe operations HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from vtprotobuf.