grpc-ecosystem/polyglot

A universal grpc command line client

polyglot's Issues

Unable to download dependencies

I could not resolve below issue. This is I am compiling polyglot from Virtual Machine where proxy is set.

No issue when I compiled in my laptop with public network.

root@BLR1000015302:/polyglot# bazel build src/main/java/me/dinowernli/grpc/polyglot
Unhandled exception thrown during build; message: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@protoc_osx_x86_64' (requested by nodes 'REPOSITORY:@protoc_osx_x86_64')
INFO: Elapsed time: 12.518s
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@protoc_osx_x86_64' (requested by nodes 'REPOSITORY:@protoc_osx_x86_64')
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:447)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:496)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at com.sun.security.ntlm.Client.type3(Client.java:161)
at sun.net.www.protocol.http.ntlm.NTLMAuthentication.buildType3Msg(NTLMAuthentication.java:241)
at sun.net.www.protocol.http.ntlm.NTLMAuthentication.setHeaders(NTLMAuthentication.java:216)
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2096)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpConnection.createAndConnectViaHttp(HttpConnection.java:144)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpConnection.createAndConnect(HttpConnection.java:92)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:149)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:92)
at com.google.devtools.build.lib.bazel.repository.HttpArchiveFunction.fetch(HttpArchiveFunction.java:66)
at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:388)
... 4 more
java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@protoc_osx_x86_64' (requested by nodes 'REPOSITORY:@protoc_osx_x86_64')
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:447)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:496)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at com.sun.security.ntlm.Client.type3(Client.java:161)
at sun.net.www.protocol.http.ntlm.NTLMAuthentication.buildType3Msg(NTLMAuthentication.java:241)
at sun.net.www.protocol.http.ntlm.NTLMAuthentication.setHeaders(NTLMAuthentication.java:216)
at sun.net.www.protocol.http.HttpURLConnection.doTunneling(HttpURLConnection.java:2096)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpConnection.createAndConnectViaHttp(HttpConnection.java:144)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpConnection.createAndConnect(HttpConnection.java:92)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:149)
at com.google.devtools.build.lib.bazel.repository.downloader.HttpDownloader.download(HttpDownloader.java:92)
at com.google.devtools.build.lib.bazel.repository.HttpArchiveFunction.fetch(HttpArchiveFunction.java:66)
at com.google.devtools.build.lib.rules.repository.RepositoryDelegatorFunction.compute(RepositoryDelegatorFunction.java:155)
at com.google.devtools.build.skyframe.ParallelEvaluator$Evaluate.run(ParallelEvaluator.java:388)
... 4 more
root@BLR1000015302:/polyglot#

Regards
-Mahesh

How to build config.proto jar?

I'm trying to create a PR about supporting grpc metadata. I changed the config.proto file but I'm unable to generate the ConfigProto.class from my change.

What's the bazel task to do it?

Support connecting over TLS

Am I being dumb?

Had some problems running grpcc (it fails only on some of our proto definitions, all of which have been successfully running in production for a while now), so I decided to try polyglot, to no avail.

[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.2.0+dev
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
: File not found.
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - Protoc invocation failed with status: 1
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: protoc version: 300, detected platform: mac os x/x86_64
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: executing: [/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/protoc2758170058438073348.exe, /Users/andrei/src/pc/proto/./portablecloud/cube/downloader/downloader.proto, -I/Users/andrei/src/pc/proto/./portablecloud/cube/downloader/downloader.proto, --descriptor_set_out=/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/descriptor2930471825732936678.pb.bin, --include_imports]
Exception in thread "main" java.lang.RuntimeException: Failed to invoke the protoc binary
	at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:96)
	at me.dinowernli.grpc.polyglot.Main.main(Main.java:52)
Caused by: me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker$ProtocInvocationException: Got exit code [1] from protoc with args [[/Users/andrei/src/pc/proto/./portablecloud/cube/downloader/downloader.proto, -I/Users/andrei/src/pc/proto/./portablecloud/cube/downloader/downloader.proto, --descriptor_set_out=/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/descriptor2930471825732936678.pb.bin, --include_imports]]
	at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invokeBinary(ProtocInvoker.java:132)
	at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invoke(ProtocInvoker.java:82)
	at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:94)
	... 1 more
➜  proto git:(master) polyglot --command=list_services --proto_discovery_root=./portablecloud/cube/downloader
[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.2.0+dev
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
gogoproto/gogo.proto: File not found.
downloader.proto: Import "gogoproto/gogo.proto" was not found or had errors.
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - Protoc invocation failed with status: 1
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: protoc version: 300, detected platform: mac os x/x86_64
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: executing: [/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/protoc766653710895866770.exe, /Users/andrei/src/pc/proto/./portablecloud/cube/downloader/downloader.proto, -I/Users/andrei/src/pc/proto/./portablecloud/cube/downloader, --descriptor_set_out=/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/descriptor1304231409264632622.pb.bin, --include_imports]
Exception in thread "main" java.lang.RuntimeException: Failed to invoke the protoc binary
	at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:96)
	at me.dinowernli.grpc.polyglot.Main.main(Main.java:52)
Caused by: me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker$ProtocInvocationException: Got exit code [1] from protoc with args [[/Users/andrei/src/pc/proto/./portablecloud/cube/downloader/downloader.proto, -I/Users/andrei/src/pc/proto/./portablecloud/cube/downloader, --descriptor_set_out=/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/descriptor1304231409264632622.pb.bin, --include_imports]]
	at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invokeBinary(ProtocInvoker.java:132)
	at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invoke(ProtocInvoker.java:82)
	at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:94)
	... 1 more```

➜ ~ echo '{"id": "vikidia-en-all@2017-04"}' | polyglot --command=call --endpoint=localhost:7007 --full_method=portablecloud.cube.downloader.Downloader.Get --proto_discovery_root=./src/pc/proto/portablecloud/cube/downloader/downloader.proto
[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.2.0+dev
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
: File not found.
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - Protoc invocation failed with status: 1
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: protoc version: 300, detected platform: mac os x/x86_64
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: executing: [/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/protoc5148216538954907554.exe, /Users/andrei/./src/pc/proto/portablecloud/cube/downloader/downloader.proto, -I/Users/andrei/./src/pc/proto/portablecloud/cube/downloader/downloader.proto, --descriptor_set_out=/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/descriptor777018564336507942.pb.bin, --include_imports]
Exception in thread "main" java.lang.RuntimeException: Failed to invoke the protoc binary
at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:96)
at me.dinowernli.grpc.polyglot.Main.main(Main.java:52)
Caused by: me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker$ProtocInvocationException: Got exit code [1] from protoc with args [[/Users/andrei/./src/pc/proto/portablecloud/cube/downloader/downloader.proto, -I/Users/andrei/./src/pc/proto/portablecloud/cube/downloader/downloader.proto, --descriptor_set_out=/var/folders/nd/ygk5rs6x1v14q_p40027lmw80000gn/T/descriptor777018564336507942.pb.bin, --include_imports]]
at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invokeBinary(ProtocInvoker.java:132)
at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invoke(ProtocInvoker.java:82)
at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:94)
... 1 more```

Add support for JSON

text format seems to be dead and for proto3 jsonpb seems to be the human-readable encoding of choice.

Please support JSON for proto3.

It should be possible to pass descriptors directly rather than running protoc

Parsing Authorization fails

Hi,
I tried to pass auth token in below format:
--metadata="Authorization:Bearer " into polyglot command line args, which results in error as space after "Bearer" is not read properly.

Is there any other way to achieve the same

Strange reponse using OneOf

Hi,

I am getting this response (array has the right size, but no fields at all):

{
  "okResponse": {
    "items": [{
    }, {
    }, {
    }]
  }
}

Using a .proto file like this (shown only relevant parts):

message ItemRpcResponse {
    int64 cat = 1;
    string s = 2;
    int32 qt = 3;
    google.protobuf.Int64Value c = 4;
    google.protobuf.Int64Value sz = 5;
    google.protobuf.Int64Value eqsz = 6;
}
message ItemsRpcResponse {
    repeated ItemRpcResponse items = 1;
}
message WrapperRpcResponse {
    oneof reply {
        ItemsRpcResponse ok_response = 1;
        RpcErrorResponse error_response = 2;
    }
}

gRPC server is returning this:

ok_response {
  items {
    id: "8c84d174-9f17-439f-af12-dbdd16e2fff3"
    cat: 1
    s: "favorite"
    qt: 1
  }
  items {
    id: "08420bd1-6f08-438c-b973-6bdaac8c6498"
    cat: 2
    s: "favorite"
    qt: 1
  }
  items {
    id: "ec3f774d-ea36-4da1-a8fe-d522770b39a7"
    cat: 3
    s: "favorite"
    qt: 1
  }
}

I am calling Polyglot v 1.6.0 Main class inside my own application, using something like this:

final InputStream stream = new ByteArrayInputStream(jsonRequest.getBytes(StandardCharsets.UTF_8));
System.setIn(stream);

final String[] polyArgs = new String[] { "--command=call", "--endpoint=" + serverAddress + ":" + port,
					"--full_method=" + serviceClass + "/" + serviceMethod, "--proto_discovery_root=" + sources.get(0),
					"--add_protoc_includes=.", "--use_tls=" + tls, "--use_reflection=false", "--deadline_ms=30000" };
me.dinowernli.grpc.polyglot.Main.main(polyArgs);

Any ideas?

Add support for $HOME/polyglot/config.json

Use proto3 JSON mapping for a config file so that OAuth2 parameters (client id, client secret, refresh token location, default paths of all generated .proto files) would be placed in there :)

redefined error

please see here protocolbuffers/protobuf#4501
thanks!

When polyglot exits due to error the exit status is 0

I've run polyglot a few times to list services or call procedures and every time it returns after a stack trace or some error the process always exists with status code 0. It would be useful if it could return with a non-zero value for the cases where it fails to perform the given operation. Of course there is also the case where we make a call to a procedure (command=call) and the RPC itself returns an error. I think in this last case it would be nice to have another flag added to polyglot where we can specify what process exit code we want it to return when the RPC itself returns with error, e.g. --rpc_err_exit_code=1 would mean "have the polyglot process exit with code 1 if RPC error != null`.

Make polyglot easy to use from scripts

Many people use \n as a separator for multiple JSON objects in a single file. I think a \n\n would work for TextFormat of proto2.

Please support mulit-stream in and out for unary and streaming gRPC services. Basically:

• read from stdin, every time you read a whole message (suing \n or \n\n as separators)
• write responses in the format of choice to stdout
• output the grpc response status as a UNIX value of the program exit (OK maps nicely to 0)
• print the response status: <grpc status>: <grpc message> to stderr if not OK
• print all exceptions and problems to stderr

This will make it easy to use polyglot in bash scripts by cat-ing output in, and processing output automatically

run call-command-example error

bratch: master
system: Mac
build cmd:
python src/tools/generate-intellij.py
bazel build src/main/java/me/dinowernli/grpc/polyglot
bash src/tools/example/run-server.sh
bash src/tools/example/call-command-example.sh

INFO: Analysed target //src/main/java/me/dinowernli/grpc/polyglot:polyglot (0 packages loaded).
INFO: Found 1 target...
Target //src/main/java/me/dinowernli/grpc/polyglot:polyglot up-to-date:
bazel-bin/src/main/java/me/dinowernli/grpc/polyglot/polyglot.jar
bazel-bin/src/main/java/me/dinowernli/grpc/polyglot/polyglot
INFO: Elapsed time: 0.156s, Critical Path: 0.00s
INFO: Build completed successfully, 1 total action
[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.7.0+dev
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration: default
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating channel to: localhost:12345
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Using proto descriptors obtained from protoc
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating dynamic grpc client
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Making rpc with 1 request(s) to endpoint [localhost:12345]
[main] INFO me.dinowernli.grpc.polyglot.grpc.DynamicGrpcClient - Making unary call
[main] INFO me.dinowernli.grpc.polyglot.grpc.DynamicGrpcClient - TEST: callUnary
[grpc-default-executor-1] ERROR me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Aborted rpc due to error
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2980246236ns
at io.grpc.Status.asRuntimeException(Status.java:526)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:385)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:61)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:504)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:425)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:536)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[main] WARN me.dinowernli.grpc.polyglot.Main - Caught top-level exception during command execution
java.lang.RuntimeException: Caught exception while waiting for rpc
at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:113)
at me.dinowernli.grpc.polyglot.Main.main(Main.java:70)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2980246236ns
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:111)
... 1 more
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2980246236ns
at io.grpc.Status.asRuntimeException(Status.java:526)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:385)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:61)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:504)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:425)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:536)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Caught exception while waiting for rpc
at me.dinowernli.grpc.polyglot.Main.main(Main.java:86)
Caused by: java.lang.RuntimeException: Caught exception while waiting for rpc
at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:113)
at me.dinowernli.grpc.polyglot.Main.main(Main.java:70)
Caused by: java.util.concurrent.ExecutionException: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2980246236ns
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:111)
... 1 more
Caused by: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 2980246236ns
at io.grpc.Status.asRuntimeException(Status.java:526)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:385)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:61)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:504)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:425)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:536)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Add integration tests for TLS

You can use the CA and TLS certificates + loaders from upstream grpc-java:

Loader:
https://github.com/grpc/grpc-java/blob/5c4817723e90b8adb8ed0cf923885df6301ed303/testing/src/main/java/io/grpc/testing/TestUtils.java#L220

Certs in here:
https://github.com/grpc/grpc-java/tree/5c4817723e90b8adb8ed0cf923885df6301ed303/testing/src/main/resources/certs

Usage:
https://github.com/grpc/grpc-java/blob/5c4817723e90b8adb8ed0cf923885df6301ed303/interop-testing/src/test/java/io/grpc/testing/integration/Http2NettyTest.java#L88

Polyglot hangs if auth fails

If exchanging the refresh token fails, Polyglot correctly prints the error, but then just hangs rather than existing.

Support add custom metadata

When extending authentication with MetadataCredentials, polyglot should support add custom metadata.

Server Reflection not used

When using Polyglot 1.6.0 it doesnt seem to trigger server reflection and always falls back to protoc this example is using the HelloWorld example from the go-grpc.

Server code with reflection: https://github.com/grpc/grpc-go/blob/master/examples/helloworld/greeter_server/main.go

Call:

echo {'name':'dom'} | java -jar ~/polyglot.jar \
--command=call \
--use_reflection=true --endpoint=localhost:50051 \
--full_method=helloworld.Greeter/SayHello \
--proto_discovery_root=.

Output:

[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.6.0
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating channel to: localhost:50051
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Using proto descriptors obtained from protoc
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating dynamic grpc client
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Making rpc with 1 request(s) to endpoint [localhost:50051]
[main] INFO me.dinowernli.grpc.polyglot.grpc.DynamicGrpcClient - Making unary call
[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Got response message
{
  "message": "Hello dom"
}

[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Completed rpc with 1 response(s)

Please release a new version

I think the addition of client cert support warrants a new release. Can you please cut a new tag and upload a new jar? Thanks!

If ServerReflection is not enabled an ERROR is logged

In 1.4.0 running the same command in previous versions we get a ERROR logged with a stack trace when the server does not have reflection turned on. The call completes successfully but would be much nicer not to have the stacktrace.

Call log:

[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.4.0
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration: default
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating channel to: 127.0.0.1:10103
[main] INFO me.dinowernli.grpc.polyglot.oauth2.OauthCredentialsFactory - Using access token credentials
[grpc-default-executor-0] ERROR me.dinowernli.grpc.polyglot.grpc.ServerReflectionClient - Error in the server reflection rpc
io.grpc.StatusRuntimeException: UNIMPLEMENTED: unknown service grpc.reflection.v1alpha.ServerReflection
	at io.grpc.Status.asRuntimeException(Status.java:543)
	at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:395)
	at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:426)
	at io.grpc.internal.ClientCallImpl.access$100(ClientCallImpl.java:76)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:512)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$700(ClientCallImpl.java:429)
	at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:544)
	at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:52)
	at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:117)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Using proto descriptors obtained from protoc
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating dynamic grpc client
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Making rpc with 1 request(s) to endpoint [127.0.0.1:10103]
[main] INFO me.dinowernli.grpc.polyglot.grpc.DynamicGrpcClient - Making unary call
[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Got response message
{
  "isOk": true
}

[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Completed rpc with 1 response(s)

Bazel integration tests pick up local config

 bazel test //src/test/java/me/dinowernli/grpc/polyglot/integration:tests

fails when there is a non-trivial ~/.polyglot/config.pb.json

Support buildng binaries which run on Windows

"no main manifest attribute, in polyglot.jar"

After building the JAR via $ bazel build src/main/java/me/dinowernli/grpc/polyglot and running via java -jar polyglot.jar ..., I get this error:

no main manifest attribute, in polyglot.jar

This is after a fresh Bazel installation on Mac with latest Java.

Capturing response

I would like to know how the output response can be written to file to convert to string format which can be manipulated for further testing.

Currently i am using polyglot as library in my project.

Support taking streams of requests as input

Work with IDE

Hello.

Which IDE do you use for working this project?
Because I'm trying to use intellij with bazel plugin, but it seems failing in loading project, so want to get more about it.

Add support for server reflection

This will allow you to drop the need for a .proto reading locally, and just rely on the remote server.

Some caching of the results may be useful.
https://github.com/grpc/grpc-go/tree/master/reflection

Not yet available for Java
grpc/grpc-java#651

Add oauth2 integration

How to supply raw byte data?

I have following (incomplete) protobuf definition

message Create {
   bytes data = 1;
}

How do I create a json request with polyglot that sets the parameter.

data=$(base64 image.jpeg)
echo '{"data" : "'$data'" }' | call --full_method=Create

doens't seem to work.

[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.2.0+dev
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration: 
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating dynamic grpc client
[main] INFO io.grpc.internal.ManagedChannelImpl - [ManagedChannelImpl@448ff1a8] Created with target localhost:8182
Exception in thread "main" java.lang.RuntimeException: Caught exception during command execution
	at me.dinowernli.grpc.polyglot.Main.main(Main.java:87)
Caused by: java.lang.IllegalArgumentException: Unable to read messages from: STDIN
	at me.dinowernli.grpc.polyglot.io.MessageReader.read(MessageReader.java:85)
	at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:75)
	at me.dinowernli.grpc.polyglot.Main.main(Main.java:72)
Caused by: com.google.protobuf.InvalidProtocolBufferException: com.google.common.io.BaseEncoding$DecodingException: Unrecognized character: 0x20
	at com.google.protobuf.util.JsonFormat$ParserImpl.merge(JsonFormat.java:992)
	at com.google.protobuf.util.JsonFormat$Parser.merge(JsonFormat.java:231)
	at me.dinowernli.grpc.polyglot.io.MessageReader.read(MessageReader.java:79)

Write message outputs to stdout rather than to the logs

Support building on windows and mac

Support streamed requests/responses

Add support for custom deadlines

Support building binaries which run on Mac

Add documentation on logging

Hi,

It would be great if you could add a paragraph about logging to the README.md file.
Something along the lines of this:

Polyglot uses the slf4j logging framework with the org.slf4j.impl.SimpleLogger implementation. It also redirects gRPC's JUL logs to slf4j. To change the default log level, specify the org.slf4j.simpleLogger.defaultLogLevel JVM argument, e.g. -Dorg.slf4j.simpleLogger.defaultLogLevel=debug.

Many thanks,
Feso

Making rpc fails if --output_file_path does not point to an existing file

Add information how to send dynamic request using polyglot

It would nice to have that clear how to invoke different kind of grpc calls like streaming client (: An example here might be useful.

Switch to using Flagz for command line flags

https://github.com/mwitkow/java-flagz

@mwitkow

Integrate with Travis CI

Write a "list" command which lists the grpc methods which have been identified

Support building on Windows

Blocked on a Bazel Windows release.

Support environment variables in config paths

Add integration tests

Distribute built binaries

"Invalid package name './src': package name component contains only '.' characters."

The README says:

Building and running tests

$ bazel test ./src/...

However, that command gives the following error (on OS X Sierra, using bazel 0.5.2):

$ bazel test ./src/...
ERROR: Invalid package name './src': package name component contains only '.' characters.
INFO: Elapsed time: 0.076s
ERROR: Couldn't start the build. Unable to run tests.

Getting rid of the leading '.' fixes the issue:

$ bazel test src/...
INFO: Found 27 targets and 8 test targets...
INFO: Elapsed time: 10.092s, Critical Path: 8.70s
//src/test/java/me/dinowernli/grpc/polyglot/command:tests                PASSED in 0.7s
//src/test/java/me/dinowernli/grpc/polyglot/config:tests                 PASSED in 0.8s
//src/test/java/me/dinowernli/grpc/polyglot/grpc:tests                   PASSED in 0.7s
//src/test/java/me/dinowernli/grpc/polyglot/integration:plain            PASSED in 5.8s
//src/test/java/me/dinowernli/grpc/polyglot/integration:tls              PASSED in 3.0s
//src/test/java/me/dinowernli/grpc/polyglot/io:tests                     PASSED in 0.9s
//src/test/java/me/dinowernli/grpc/polyglot/oauth2:tests                 PASSED in 0.5s
//src/test/java/me/dinowernli/grpc/polyglot/protobuf:tests               PASSED in 0.8s

Executed 8 out of 8 tests: 8 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

I realize I could submit a PR, but for a README tweak it seems like overkill...

Java 7 support

Are there any parts of polyglot that heavily depend on Java 8?

I need Java 7 support for the project I am working on.

Feature Request - Cache generated clients / descriptors

Would be good to be able to cache some of the compiled protos or clients that are created so that they can be used again if the .proto file has not been altered. This would hopefully speed up subsequent calls to polyglot.

First call:

time echo {'name':'dom1'} | java -jar ~/polyglot.jar \
--command=call \
--endpoint=localhost:50051 \
--full_method=helloworld.Greeter/SayHello \
--proto_discovery_root=/Users/dominicgreen/go/src/google.golang.org/grpc/examples/helloworld/helloworld
[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.6.0
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating channel to: localhost:50051
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Using proto descriptors obtained from protoc
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating dynamic grpc client
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Making rpc with 1 request(s) to endpoint [localhost:50051]
[main] INFO me.dinowernli.grpc.polyglot.grpc.DynamicGrpcClient - Making unary call
[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Got response message
{
  "message": "Hello dom1"
}

[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Completed rpc with 1 response(s)
echo {'name':'dom1'}  0.00s user 0.00s system 47% cpu 0.003 total
java -jar ~/polyglot.jar --command=call --endpoint=localhost:50051    1.74s user 0.18s system 30% cpu 6.403 total

Second call, calls protoc again and takes similar time:

time echo {'name':'dom2'} | java -jar ~/polyglot.jar \
--command=call \
--endpoint=localhost:50051 \
--full_method=helloworld.Greeter/SayHello \
--proto_discovery_root=/Users/dominicgreen/go/src/google.golang.org/grpc/examples/helloworld/helloworld
[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.6.0
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating channel to: localhost:50051
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Using proto descriptors obtained from protoc
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Creating dynamic grpc client
[main] INFO me.dinowernli.grpc.polyglot.command.ServiceCall - Making rpc with 1 request(s) to endpoint [localhost:50051]
[main] INFO me.dinowernli.grpc.polyglot.grpc.DynamicGrpcClient - Making unary call
[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Got response message
{
  "message": "Hello dom2"
}

[grpc-default-executor-0] INFO me.dinowernli.grpc.polyglot.io.LoggingStatsWriter - Completed rpc with 1 response(s)
echo {'name':'dom2'}  0.00s user 0.00s system 39% cpu 0.004 total
java -jar ~/polyglot.jar --command=call --endpoint=localhost:50051    1.99s user 0.21s system 32% cpu 6.694 total

Cope with TLS client keys in PKCS#1 format

If you try to connect with --tls_client_key_path point to a valid private key in PKCS#1 format, Polyglot dies with:

java.lang.IllegalArgumentException: File does not contain valid private key: out/polyglot.key
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:222)
        at me.dinowernli.grpc.polyglot.grpc.ChannelFactory.createSslContext(ChannelFactory.java:105)
        at me.dinowernli.grpc.polyglot.grpc.ChannelFactory.createChannelBuilder(ChannelFactory.java:71)
        at me.dinowernli.grpc.polyglot.grpc.ChannelFactory.createChannel(ChannelFactory.java:51)
        at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:68)
        at me.dinowernli.grpc.polyglot.Main.main(Main.java:70)
Caused by: java.security.NoSuchAlgorithmException: EC KeyFactory not available
        at java.security.KeyFactory.<init>(java.base@9-internal/KeyFactory.java:138)
        at java.security.KeyFactory.getInstance(java.base@9-internal/KeyFactory.java:179)
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)
        ... 6 more
Exception in thread "main" java.lang.RuntimeException: java.lang.IllegalArgumentException: File does not contain valid private key: out/polyglot.key
        at me.dinowernli.grpc.polyglot.Main.main(Main.java:86)
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: out/polyglot.key
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:222)
        at me.dinowernli.grpc.polyglot.grpc.ChannelFactory.createSslContext(ChannelFactory.java:105)
        at me.dinowernli.grpc.polyglot.grpc.ChannelFactory.createChannelBuilder(ChannelFactory.java:71)
        at me.dinowernli.grpc.polyglot.grpc.ChannelFactory.createChannel(ChannelFactory.java:51)
        at me.dinowernli.grpc.polyglot.command.ServiceCall.callEndpoint(ServiceCall.java:68)
        at me.dinowernli.grpc.polyglot.Main.main(Main.java:70)
Caused by: java.security.NoSuchAlgorithmException: EC KeyFactory not available
        at java.security.KeyFactory.<init>(java.base@9-internal/KeyFactory.java:138)
        at java.security.KeyFactory.getInstance(java.base@9-internal/KeyFactory.java:179)
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)
        ... 6 more

Converting the key with this command line makes it work:

openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in polyglot.key -out polyglot-pkcs8.key

Would be nice if Polyglot figured this out on its own or at least told you what the issue is.

Unable to call with an 'Any' type proto.

When using the Any type when making a polyglot server call I hit InvalidProtocolBufferException: Cannot resolve type: type.googleapis.com/my.thingie

Example, the file I pipe to polyglot contains:

{"anyTypefield":{"@type":"type.googleapis.com/my.thingie","value":"Ig5mb29BY2NvdW50TmFtZQ=="}}

my.thingie is well defined and within the --add_protoc_includes.

The proto spec is:

message thingie 
  google.protobuf.Any anyTypefield = 1;
}

The type URL "type.googleapis.com/my.proto.type" is the one produced by (jsonpb.Marshaler{}).MarshalToString() under golang, so expecting this to be the correct type reference.

When I use my.thingie as a normal type (not within an Any field), polyglot is able to see and use the type without issues.

Using polyglot as a library

I would like to use polyglot as a library e.g. ideally pulling in a dependency from central that has all the transitive dependencies declared in a pom.xml.

Currently, I am using a local repo with a modified release JAR with the JVM dependencies ripped out and a manually crafted pom.xml used to pull in the JVM dependencies but this is not ideal.

What is the current suggested approach for this use case? Or is polyglot only intended to be used as a standalone binary?

Thanks!

polyglot jar not including Well Known Types for protos

The released jar for polyglot is not working with well known types for protos (i.e. timestamps).

protoc run locally works fine and the well known types are present in /usr/local/include/google/protobuf/

Michaels-MacBook-Pro:idls sindelar$ java -jar polyglot.jar --command=list_services --proto_discovery_root=model/
[main] INFO me.dinowernli.grpc.polyglot.Main - Polyglot version: 1.3.0
[main] INFO me.dinowernli.grpc.polyglot.Main - Loaded configuration:
google/protobuf/timestamp.proto: File not found.
replica.proto: Import "google/protobuf/timestamp.proto" was not found or had errors.
replica.proto:120:3: "google.protobuf.Timestamp" is not defined.
replica.proto:122:3: "google.protobuf.Timestamp" is not defined.
replica.proto:145:3: "google.protobuf.Timestamp" is not defined.
replica.proto:146:3: "google.protobuf.Timestamp" is not defined.
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - Protoc invocation failed with status: 1
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: protoc version: 300, detected platform: mac os x/x86_64
[main] WARN me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker - [Protoc log] protoc-jar: executing: [/var/folders/xz/ytdp_wr52cj2jhxz76_m6b6w0000gn/T/protoc4924784953496390184.exe, /Users/sindelar/sidewalk/idls/model/replica.proto, /Users/sindelar/sidewalk/idls/model/map_service.proto, /Users/sindelar/sidewalk/idls/model/persona.proto, -I/Users/sindelar/sidewalk/idls/model, --descriptor_set_out=/var/folders/xz/ytdp_wr52cj2jhxz76_m6b6w0000gn/T/descriptor904114720190676395.pb.bin, --include_imports]
Exception in thread "main" java.lang.RuntimeException: Failed to invoke the protoc binary
	at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:96)
	at me.dinowernli.grpc.polyglot.Main.main(Main.java:52)
Caused by: me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker$ProtocInvocationException: Got exit code [1] from protoc with args [[/Users/sindelar/sidewalk/idls/model/replica.proto, /Users/sindelar/sidewalk/idls/model/map_service.proto, /Users/sindelar/sidewalk/idls/model/persona.proto, -I/Users/sindelar/sidewalk/idls/model, --descriptor_set_out=/var/folders/xz/ytdp_wr52cj2jhxz76_m6b6w0000gn/T/descriptor904114720190676395.pb.bin, --include_imports]]
	at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invokeBinary(ProtocInvoker.java:132)
	at me.dinowernli.grpc.polyglot.protobuf.ProtocInvoker.invoke(ProtocInvoker.java:82)
	at me.dinowernli.grpc.polyglot.Main.getFileDescriptorSet(Main.java:94)
	... 1 more

Seems related to this issue
protocolbuffers/protobuf#1889