Figure out how our users will use the protoc plugin

[ejona86]

Figure out how our users will use the protoc plugin

[zhangkun83]

We have considered the compiler from sqaure/wire as a potential option, but it turns out not as appealing as we thought:

1. It generates its own implementation of proto messages (Message), although we want to stick with the standard protobuf implementation. We would have to rewrite the message codegen in Java, which is non-trivial and would become a maintenance burden.
2. The parser and the codgen are linked together. It's not as flexible as protoc, because protoc emits results in the form of descriptor proto to the codegen plugin, which makes it possible to write the plugin in any language.

[zhangkun83]

As protoc emits parsed definitions in the form of descriptor proto, it's possible to write the plugin in Java. I am not sure how much better it looks like than our current solution, but at least we won't have C++ code in grpc-java tree.

Update: Probably not a good idea. The plugin needs to figure out the proper package of message classes, which is currently done by helper functions in C++ and we won't want to duplicate it in Java.

[ejona86]

This would be super-nice for 0.5.0, but I don't actually expect it to happen in time.

[He-Pin]

we are using sbt-protobuf plugin

[zhangkun83]

We (Eric and I) have looked at multiple approaches to serve native plugin artifact from a repo:

1. Packaging as a zip and unzip it at the user-end.
2. Packaging as a nar and allow Maven user to download and unpack it with maven-nar-plugin
3. Uploading the binary artifact directly as .exe file, and modify gradle-plugin-protobuf and maven-protoc-plugin to accept dependencies on native plugin artifacts and use them directly.

We have decided to go with option 3, because additional packaging doesn't sound necessary. We will also need to have a uniform format of classifiers of the artifact, because each OS/architecture will have a separate artifact. We will use os-maven-plugin, which Netty has been using, to generate the classifiers. I have worked out a plugin for Gradle that generates identical results.

[sergei-ivanov]

@zhangkun83 This is a great summary, it looks like you've spent a great deal of time researcing the options. I still think that a pure java protoc plugin would be great, especially that we have had a properly working support for pure java plugins for quite some time, and it worked really well in our case.
There is a [rather trivial] example bundled in maven plugin ITs:
https://github.com/sergei-ivanov/maven-protoc-plugin/tree/master/src/it/setup-protoc-plugin
https://github.com/sergei-ivanov/maven-protoc-plugin/tree/master/src/it/TEST-16

However, if you think a pure java plugin would result in a massive maintenance burden, I understand your decision to stick to native binaries. Option 3 above of uploading a bunch of binaries compiled for different os/arch combinations looks good to me and will certainly save a lot of effort. Hopefully the main protobuf project will do the same for protoc. I shall see how all that can be accommodated into the maven plugin.

[zhangkun83]

osdetector-gradle-plugin is now available on Maven Central. I will be working on forking gradle-plugin-protobuf and pushing native protoc and GRPC plugin to Maven Central.

[ejona86]

@sergei-ivanov, one of the parts of option 3 is that we are getting binaries for protoc supported by protobuf project and in Maven Central (!!). We felt we really needed to solve the problem of protoc distribution. Since solving protoc distribution is almost the same as our binary plugin, we don't gain much (in the short-term) by swapping to a Java protoc plugin. We may still swap to a Java protoc plugin in the future, but short-term it didn't gain us much.

[sergei-ivanov]

@ejona86 Yes, it is more like wishful thinking on my part. I really wish protoc was written in Java to start with, because that would have solved all portability problems at once. But we've got what we've got, and having binaries in central is better than having nothing at all. I hope your Linux binaries are going to run under not-so-modern kernel/glibc versions too, because in sectors like finance Linux installations will typically be 3-5 years behind the latest stable.

[zhangkun83]

@sergei-ivanov Our 32-bit linux binaries will have following dependencies:

• linux-gate.so.1
• libpthread.so.0
• libm.so.6
• libc.so.6
• /lib/ld-linux.so.2

Is it good to run on the not-so-modern systems of your concern?

[sergei-ivanov]

@zhangkun83 Thanks for the information, I'll find it out, but unfortunately not earlier than the next week. Good Friday and Easter Monday are public holidays here in the UK, and I'll need to poll my contacts in other UK banks too (protobufs have become quite popular in the finance sector in the last few years).

[sergei-ivanov]

@zhangkun83
I have just checked our build server, and it has got 4 out of 5 required libraries.

$ ls -l /lib | grep -e '\(libpthread.so.0\)\|\(linux-gate.so.1\)\|\(libm.so.6\)\|\(libc.so.6\)\|\(ld-linux.so.2\)'
lrwxrwxrwx 1 root root       9 Mar 17  2014 ld-linux.so.2 -> ld-2.5.so
lrwxrwxrwx 1 root root      11 Mar 17  2014 libc.so.6 -> libc-2.5.so
lrwxrwxrwx 1 root root      11 Mar 17  2014 libm.so.6 -> libm-2.5.so
lrwxrwxrwx 1 root root      17 Mar 17  2014 libpthread.so.0 -> libpthread-2.5.so

Similar results for 64-bit libs (ld-linux.so has an extra suffix):

$ ls -l /lib64 | grep -e '\(libpthread.so.0\)\|\(linux-gate.so.1\)\|\(libm.so.6\)\|\(libc.so.6\)\|\(ld-linux.*.so.2\)'
lrwxrwxrwx 1 root root       9 Mar 17  2014 ld-linux-x86-64.so.2 -> ld-2.5.so
lrwxrwxrwx 1 root root      11 Mar 17  2014 libc.so.6 -> libc-2.5.so
lrwxrwxrwx 1 root root      11 Mar 17  2014 libm.so.6 -> libm-2.5.so
lrwxrwxrwx 1 root root      17 Mar 17  2014 libpthread.so.0 -> libpthread-2.5.so

However, linux-gate could not be found anywhere on the file system, the following command returns nothing:

$ find / -name 'linux-gate.*' 2>&1 | grep -v 'Permission denied' | grep -v 'Filesystem loop detected'

Linux version:

$ uname -a
Linux xxxxxxxxxx 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[zhangkun83]

@sergei-ivanov linux-gate.so.1 (as well as linux-vdso.so.1 on 64-bit systems) does not exist on the file system. So it looks we are good.

[sergei-ivanov]

@zhangkun83 I've just taken another look at dynamically resolving Maven artifacts by GAV, and artifact type appears to be a required parameter in Maven, because it determines the artifact handler to be used. In the light of that, I think it would make sense to upload the protoc and grpc-plugin artifacts with the same type, e.g. exe, for all flavours. I realise that exe sounds silly for Linux/Solaris/MacOS environments, but at least it will make Maven plugin configuration simpler, because with a common GAV part, the os/arch-specific classifier will be the only variable part in artifact identification and resolution.

[zhangkun83]

This is exactly what we are doing. The protoc artifact will be named like protoc-3.0.0-alpha2-linux-x86_64.exe

[sergei-ivanov]

That is great, makes life a bit simpler on my side as well. Thanks for the confirmation.

Sergei

[zhangkun83]

I have pushed our new protobuf-gradle-plugin to Maven Central. It can pull protoc and protoc-gen-grpc-java from repositories. I will push protoc artifacts to Maven Central soon.

[sergei-ivanov]

Thanks! I've got Maven plugin almost ready too, but I think I'll wait until protoc and grpc binaries are in central, because then I'll be able to reference the new artifacts from the integration tests.

[zhangkun83]

@sergei-ivanov check out the protoc artifacts on Central!
GRPC codegen artifacts will be released with the upcoming grpc-java 0.5.0 release.

[zhangkun83]

@sergei-ivanov We have been pushing grpc codegen snapshots to Sonatype snapshot repository. You can test your Maven plugin by pointing your repo to https://oss.sonatype.org/content/repositories/snapshots/ and reference groupId: io.grpc, artifactId: protoc-gen-grpc-java, version: 0.1.0-SNAPSHOT.

[ejona86]

This was resolved with the release on May 26th, combined with the work done for the Gradle and Maven plugins.