Comments (21)
We have considered the compiler from sqaure/wire as a potential option, but it turns out not as appealing as we thought:
- It generates its own implementation of proto messages (Message), although we want to stick with the standard protobuf implementation. We would have to rewrite the message codegen in Java, which is non-trivial and would become a maintenance burden.
- The parser and the codgen are linked together. It's not as flexible as protoc, because protoc emits results in the form of descriptor proto to the codegen plugin, which makes it possible to write the plugin in any language.
from grpc-java.
As protoc emits parsed definitions in the form of descriptor proto, it's possible to write the plugin in Java. I am not sure how much better it looks like than our current solution, but at least we won't have C++ code in grpc-java tree.
Update: Probably not a good idea. The plugin needs to figure out the proper package of message classes, which is currently done by helper functions in C++ and we won't want to duplicate it in Java.
from grpc-java.
This would be super-nice for 0.5.0, but I don't actually expect it to happen in time.
from grpc-java.
we are using sbt-protobuf plugin
from grpc-java.
We (Eric and I) have looked at multiple approaches to serve native plugin artifact from a repo:
- Packaging as a zip and unzip it at the user-end.
- Packaging as a nar and allow Maven user to download and unpack it with maven-nar-plugin
- Uploading the binary artifact directly as
.exe
file, and modify gradle-plugin-protobuf and maven-protoc-plugin to accept dependencies on native plugin artifacts and use them directly.
We have decided to go with option 3, because additional packaging doesn't sound necessary. We will also need to have a uniform format of classifiers of the artifact, because each OS/architecture will have a separate artifact. We will use os-maven-plugin, which Netty has been using, to generate the classifiers. I have worked out a plugin for Gradle that generates identical results.
from grpc-java.
@zhangkun83 This is a great summary, it looks like you've spent a great deal of time researcing the options. I still think that a pure java protoc plugin would be great, especially that we have had a properly working support for pure java plugins for quite some time, and it worked really well in our case.
There is a [rather trivial] example bundled in maven plugin ITs:
https://github.com/sergei-ivanov/maven-protoc-plugin/tree/master/src/it/setup-protoc-plugin
https://github.com/sergei-ivanov/maven-protoc-plugin/tree/master/src/it/TEST-16
However, if you think a pure java plugin would result in a massive maintenance burden, I understand your decision to stick to native binaries. Option 3 above of uploading a bunch of binaries compiled for different os/arch combinations looks good to me and will certainly save a lot of effort. Hopefully the main protobuf project will do the same for protoc. I shall see how all that can be accommodated into the maven plugin.
from grpc-java.
osdetector-gradle-plugin is now available on Maven Central. I will be working on forking gradle-plugin-protobuf and pushing native protoc and GRPC plugin to Maven Central.
from grpc-java.
@sergei-ivanov, one of the parts of option 3 is that we are getting binaries for protoc supported by protobuf project and in Maven Central (!!). We felt we really needed to solve the problem of protoc distribution. Since solving protoc distribution is almost the same as our binary plugin, we don't gain much (in the short-term) by swapping to a Java protoc plugin. We may still swap to a Java protoc plugin in the future, but short-term it didn't gain us much.
from grpc-java.
@ejona86 Yes, it is more like wishful thinking on my part. I really wish protoc was written in Java to start with, because that would have solved all portability problems at once. But we've got what we've got, and having binaries in central is better than having nothing at all. I hope your Linux binaries are going to run under not-so-modern kernel/glibc versions too, because in sectors like finance Linux installations will typically be 3-5 years behind the latest stable.
from grpc-java.
@sergei-ivanov Our 32-bit linux binaries will have following dependencies:
- linux-gate.so.1
- libpthread.so.0
- libm.so.6
- libc.so.6
- /lib/ld-linux.so.2
Is it good to run on the not-so-modern systems of your concern?
from grpc-java.
@zhangkun83 Thanks for the information, I'll find it out, but unfortunately not earlier than the next week. Good Friday and Easter Monday are public holidays here in the UK, and I'll need to poll my contacts in other UK banks too (protobufs have become quite popular in the finance sector in the last few years).
from grpc-java.
@zhangkun83
I have just checked our build server, and it has got 4 out of 5 required libraries.
$ ls -l /lib | grep -e '\(libpthread.so.0\)\|\(linux-gate.so.1\)\|\(libm.so.6\)\|\(libc.so.6\)\|\(ld-linux.so.2\)'
lrwxrwxrwx 1 root root 9 Mar 17 2014 ld-linux.so.2 -> ld-2.5.so
lrwxrwxrwx 1 root root 11 Mar 17 2014 libc.so.6 -> libc-2.5.so
lrwxrwxrwx 1 root root 11 Mar 17 2014 libm.so.6 -> libm-2.5.so
lrwxrwxrwx 1 root root 17 Mar 17 2014 libpthread.so.0 -> libpthread-2.5.so
Similar results for 64-bit libs (ld-linux.so
has an extra suffix):
$ ls -l /lib64 | grep -e '\(libpthread.so.0\)\|\(linux-gate.so.1\)\|\(libm.so.6\)\|\(libc.so.6\)\|\(ld-linux.*.so.2\)'
lrwxrwxrwx 1 root root 9 Mar 17 2014 ld-linux-x86-64.so.2 -> ld-2.5.so
lrwxrwxrwx 1 root root 11 Mar 17 2014 libc.so.6 -> libc-2.5.so
lrwxrwxrwx 1 root root 11 Mar 17 2014 libm.so.6 -> libm-2.5.so
lrwxrwxrwx 1 root root 17 Mar 17 2014 libpthread.so.0 -> libpthread-2.5.so
However, linux-gate
could not be found anywhere on the file system, the following command returns nothing:
$ find / -name 'linux-gate.*' 2>&1 | grep -v 'Permission denied' | grep -v 'Filesystem loop detected'
Linux version:
$ uname -a
Linux xxxxxxxxxx 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
from grpc-java.
@sergei-ivanov linux-gate.so.1
(as well as linux-vdso.so.1
on 64-bit systems) does not exist on the file system. So it looks we are good.
from grpc-java.
@zhangkun83 I've just taken another look at dynamically resolving Maven artifacts by GAV, and artifact type appears to be a required parameter in Maven, because it determines the artifact handler to be used. In the light of that, I think it would make sense to upload the protoc
and grpc-plugin
artifacts with the same type, e.g. exe
, for all flavours. I realise that exe
sounds silly for Linux/Solaris/MacOS environments, but at least it will make Maven plugin configuration simpler, because with a common GAV part, the os/arch-specific classifier will be the only variable part in artifact identification and resolution.
from grpc-java.
This is exactly what we are doing. The protoc artifact will be named like protoc-3.0.0-alpha2-linux-x86_64.exe
from grpc-java.
That is great, makes life a bit simpler on my side as well. Thanks for the confirmation.
Sergei
from grpc-java.
I have pushed our new protobuf-gradle-plugin to Maven Central. It can pull protoc
and protoc-gen-grpc-java
from repositories. I will push protoc
artifacts to Maven Central soon.
from grpc-java.
Thanks! I've got Maven plugin almost ready too, but I think I'll wait until protoc
and grpc
binaries are in central, because then I'll be able to reference the new artifacts from the integration tests.
from grpc-java.
@sergei-ivanov check out the protoc
artifacts on Central!
GRPC codegen artifacts will be released with the upcoming grpc-java 0.5.0
release.
from grpc-java.
@sergei-ivanov We have been pushing grpc codegen snapshots to Sonatype snapshot repository. You can test your Maven plugin by pointing your repo to https://oss.sonatype.org/content/repositories/snapshots/
and reference groupId: io.grpc, artifactId: protoc-gen-grpc-java, version: 0.1.0-SNAPSHOT
.
from grpc-java.
This was resolved with the release on May 26th, combined with the work done for the Gradle and Maven plugins.
from grpc-java.
Related Issues (20)
- What does "length" means in i.g.n.NettyClientHandler INBOUND DATA HOT 2
- GRPC Java MAX_CONCURRENT_STREAMS vs maxConcurrentCallsPerConnection HOT 1
- BinderChannel: allow kotlinx.serialization as protobuf alternative HOT 2
- Found one potential memory leak logic
- Examples fail to build on JDK 22 HOT 2
- Track OpenTelemetry Tracing being experimental
- connection reset using firebase firestore using Php and Html HOT 1
- Add a global TypeRegistry, similar to the Go implementation HOT 1
- MultiChildLb unconditionally calls requestConnection() HOT 4
- IS_PETIOLE_POLICY is not used by PickFirstLeafLoadBalancer
- Pumping to version 1.65.2 from version 1.62.2 gave the following error 'failed with code: BACKEND_FAILURE_URL_UNREACHABLE' HOT 1
- I can't run the android client example. HOT 2
- Unexpected INTERNAL error io.grpc.StatusRuntimeException: INTERNAL: http2 exception HOT 2
- Header size exceeded max allowed size (10240) HOT 6
- xds: Incorrect load reporting when using pick first as locality-routing policy
- Got ClassNotFoundException while building native image with grpc-netty-shaded HOT 1
- Android grpc with UDS is slow to connect sometimes HOT 6
- Add CI build with netty 4.2.0.Alpha snapshots / releases HOT 10
- monitoring component for related performance indicators (such as rt qps error rate, etc.) HOT 1
- Release notes for 1.66.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from grpc-java.