grpc/grpc-go

The Go language implementation of gRPC. HTTP/2 based RPC

gRPC-Go

The Go implementation of gRPC: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information see the Go gRPC docs, or jump directly into the quick start.

Prerequisites

• Go: any one of the three latest major releases.

Installation

Simply add the following import to your code, and then go [build|run|test] will automatically fetch the necessary dependencies:

import "google.golang.org/grpc"

Note: If you are trying to access grpc-go from China, see the FAQ below.

Learn more

• Go gRPC docs, which include a quick start and API reference among other resources
• Low-level technical docs from this repository
• Performance benchmark
• Examples

FAQ

I/O Timeout Errors

The golang.org domain may be blocked from some countries. go get usually produces an error like the following when this happens:

$ go get -u google.golang.org/grpc
package google.golang.org/grpc: unrecognized import path "google.golang.org/grpc" (https fetch: Get https://google.golang.org/grpc?go-get=1: dial tcp 216.239.37.1:443: i/o timeout)

To build Go code, there are several options:

• Set up a VPN and access google.golang.org through that.

• With Go module support: it is possible to use the replace feature of go mod to create aliases for golang.org packages. In your project's directory:

  go mod edit -replace=google.golang.org/grpc=github.com/grpc/grpc-go@latest
  go mod tidy
  go mod vendor
  go build -mod=vendor

  Again, this will need to be done for all transitive dependencies hosted on golang.org as well. For details, refer to golang/go issue #28652.

Compiling error, undefined: grpc.SupportPackageIsVersion

Please update to the latest version of gRPC-Go using go get google.golang.org/grpc.

How to turn on logging

The default logger is controlled by environment variables. Turn everything on like this:

$ export GRPC_GO_LOG_VERBOSITY_LEVEL=99
$ export GRPC_GO_LOG_SEVERITY_LEVEL=info

The RPC failed with error "code = Unavailable desc = transport is closing"

This error means the connection the RPC is using was closed, and there are many possible reasons, including:

1. mis-configured transport credentials, connection failed on handshaking
2. bytes disrupted, possibly by a proxy in between
3. server shutdown
4. Keepalive parameters caused connection shutdown, for example if you have configured your server to terminate connections regularly to trigger DNS lookups. If this is the case, you may want to increase your MaxConnectionAgeGrace, to allow longer RPC calls to finish.

It can be tricky to debug this because the error happens on the client side but the root cause of the connection being closed is on the server side. Turn on logging on both client and server, and see if there are any transport errors.

grpc-go's Issues

assumes host:port style addresses, cannot easily use custom transports, e.g. unix domain sockets

https://github.com/grpc/grpc-go/blob/master/transport/http2_client.go#L106-L119

grpc.Dial spins forever if target address is bad

Specifically, resetTransport discards any underlying error from transport.NewClientTransport. Should there be a configurable limit on retries? If the service I'm talking to has gone away, it would be nice to not spin forever.

grpc.Code(grpc.Errorf(codes.OK, ...)) returns codes.Unknown instead of codes.OK

grpc.Errorf(codes.OK, ...) is documented to return nil. Could grpc.Code be changed to check for nil and return codes.OK, instead of codes.Unknown?

Stream doesn't know that transport is broken

How to reproduce:

1. Do normal server streaming rpc
2. Streaming runs smoothly
3. Turn off network
4. ClientStream.RecvMsg() waits indefinitely, there's no way to cancel
   ServerStream.SendMsg() successfully sending (nil error) until, I assume, transport.defaultWindowSize, there's no way to cancel()
   This happens no matter how long I turn off the network.

Expectation:

1. Provide a way to cancel method instead of waiting indefinitely.
2. Stream should be aware of broken transport in the middle of transmission

I provide a gist to help reproduce the problem

Concurrency

How does grpc handle concurrency? Can I use a client from different threads in parallel?

context.Context on the server side

I see that we are to provide a context.Context on the client side:

https://github.com/grpc/grpc-go/blob/master/examples/route_guide/client/client.go#L73

However, should the context.Context be made available on the server side as well?

http://talks.golang.org/2014/gotham-context.slide#13

server checks :authority header in the secure channel

We allow users to override the authority per-call, but we currently don't do any verification that that authority would be permitted for the current server. We should verify the provided authority against the TLS cert of the connection and fail in some way if the cert is not good for the requested authority. We would cache these verifications for the connection in a simple hash map.

It is the Java equivalent of grpc/grpc#471

I/O contention caused by unbuffered net.Conn read/write.

Using Go v1.4.2.

@codahale was seeing low numbers (~12K req/sec) while benchmarking grpc performance between two EC2 instances. He generated a CPU profile, and we realized that an inordinate amount of time was being spent in syscall.Syscall, which looked to us like IO contention, given the subgraph of calls leading up to it.

Digging into bradfitz/http2, we realized that http2.NewFramer accepts io.Writer and io.Reader interfaces, and does not do any buffering on its own. It's called in transport.newHTTP2Server using the underlying net.Conn for both the io.Writer and io.Reader parameters.

Per the documentation, net.Conn doesn't provide any buffering, and it appears that other packages, such as http do their own buffering using bufio.

I'm toying with the idea of submitting a PR for this that uses a technique similar to http's to reuse bufio.Writers and bufio.Readers.

no way to set background context of rpc server

There doesn't seem to be a way to control the root context of the server. For example, it would be nice to have at least something like the following:

ctx := MySpecialContext()
s := NewServer(WithContext(ctx))

Such that the context passed to an implemented rpc method descends from the above:

func (mg *myGreeter) SayHello(context.Context, *HelloRequest) (*HelloReply, error) {}

Following from that, there doesn't seem to be a way to "prepare" a context using headers. One could imagine a per-request context preparation that incorporates header metadata (ie spans, auth, metrics, etc.) at the entire sever-level.

I may be missing API details in the documentation.

Don't see a way to know the current state of a client connection (connected or not)

I am trying to use grpc as the transport for exchanging messages on a cluster.

I don't have a "service discovery" yet, so I am starting with hard-coded cluster peers. When one peer start it will create a grpc service and then open one client connection to each other peer.

When the cluster is starting up, some of the client connections will be pending, waiting for the remote peer to start. The documentation seems to imply the client Dial should fail, but instead it retries in the background. This is what I want, but I don't think I have any way to know when the connection succeed so that I can start sending requests.

I guess I could use the WithTimeout(t) option on the Dial, and do my own retries, but it would be great if there was a way to just check the client object. The other alternative is to use a context with timeout when calling the various rpc methods (with context.Background all requests seem to wait a few seconds before aborting) but I don't really want to wait if the connection is not active yet nor I want to start a goroutine for each rpc call (I need to preserve ordering).

Naming: generating a proto service should generate an interface ending in "Service" not "Server"

When generating the the 'pb.proto' file after defining a service in the protofile. The interface for the service is named 'Server' but it should be named 'Service' to keep consistency A grpc server can have several services registered.

Instead of having to call

pb.RegisterMyNameServer(*grpc.Server,MyNameServer) 

it should be

pb.RegisterMyNameService(*grpc.Server,MyNameService) 

The rest of the 'pb.proto' file is consistent afaik.

Dial WithTimeout prevents client from reconnecting

Setting the grpc.WithTimeout option in grpc.Dial causes all RPCs to fail with the error "grpc: the client connection is closing" after a server restart.

Steps to reproduce:

1. Create a client with a connection timeout:

   conn, err := grpc.Dial(address, grpc.WithTimeout(timeout))

2. Make an RPC call. It succeeds.

3. Shut down the server

4. Make an RPC call with a deadline. As expected, it cannot connect and fails:
   'rpc error: code = 4 desc = "context deadline exceeded"'

5. Start up the server

6. Make another RPC call. I expect this to work. Instead, it and each subsequent RPC fails with
   'rpc error: code = 2 desc = "grpc: the client connection is closing"'

go vet says Errorf calls are all wrong, "constant 1 not a string in call to Errorf"

go vet seems assume that Errorf first arg would be format string

$ cat main.go 
package main

import (
    "google.golang.org/grpc"
    "google.golang.org/grpc/codes"
)

func main() {
    _ = grpc.Errorf(codes.Canceled, "i have nothing more to say")
}
$ go vet main.go 
main.go:9: constant 1 not a string in call to Errorf
exit status 1

Support serving web content from the same port

https://github.com/grpc/grpc-common/blob/master/PROTOCOL-HTTP2.md#appendix-a---grpc-for-protobuf says grpc protobuf mapping uses service names as paths. Would it be possible to serve web content from other urls?

I see TLS side does alpn, so that's an easy place to hook up.

Any thoughts about non-TLS? e.g. a service running on localhost. Of course that would mean needing to do a http/1 upgrade negotiation, as then the port could not default to http/2.

Use case 1: host a web application and its api on the same port.

Use case 2: serve a health check response.

Use case 3: serve a "nothing to see here" html page.

Use case 4: serve a /robots.txt file.

Simple password authentication example

I'm trying to figure out how to authenticate requests in simple password based/session token case.

Should I :

• create my own proto, embedding token in each message ? or
• do it in grpc: implementing my own credential.Credentials ? how to check server side ?

Would appreciate if there's some basic examples.

grpc-java channel similars in grpc-go

Hi, @iamqizhao

Sorry to raise the old issue again. I think I understand the problem better. We actually want something like what grpc-java channel provides:

https://github.com/grpc/grpc-java#channel

This is very useful for framework developers.

Basically, the framework should be agnostic to message definition by user. It needs to handles the low level details, e.g. creating connection, finding address, flow control.

Ultimately, some generic client method calls is wanted

Client.Call( methodName, input message)

And interceptor method on server side

Intercept( methodName, input )

Can you think about it and let me know your thoughts?

rpc interceptor

Application might want to audit the RPC calls invoked by the clients and received by the server.
Application might want to do global rate limiting or synchronization.

It is a little bit tedious to add the instruments/synchronization code into every service grpc generated. It would be better if the application can have a centralized place to do it.

At the client side, we can actually wrapper the client interface with a thin layer. But we still cannot make this happen dynamically at runtime.

At the server side, there is no clean way that I am aware of to do it right now.

Is this a problem grpc-go would like to solve or it should be solved somewhere else? Do you guys have any suggestions?

Awesome work on grpc and thanks a lot in advance!

client fails annoyingly with lots of log messages when server does not speak grpc

I am running "go version go1.4.1 darwin/amd64". I accidentally pointed a grpc client at an address that didn't speak grpc. Afterwards, grpc printed a lot of messages in the log that were not helpful at best and distracting at worst. I would prefer grpc to a) not generate as many errors, perhaps with some back-off mechanism, and b) not print as many errors.

Specifically, my terminal filled with hundreds of lines of the form

2015/03/16 21:08:01 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.
2015/03/16 21:08:01 transport: http2Client.notifyError got notified that the client transport was broken unexpected EOF.

I tried sticking in a "c.failFast = true" in grpc.Invoke, but that did not help.

Application cannot return errors with `gprc.Code(err) != codes.Unknown`, errors are noisy, codes have no Stringer

$ cd $GOPATH/src/github.com/grpc/grpc-common/go
$ git diff
diff --git i/go/greeter_server/main.go w/go/greeter_server/main.go
index c7fa06a..c7682cb 100644
--- i/go/greeter_server/main.go
+++ w/go/greeter_server/main.go
@@ -40,6 +40,7 @@ import (
        pb "github.com/grpc/grpc-common/go/helloworld"
        "golang.org/x/net/context"
        "google.golang.org/grpc"
+       "google.golang.org/grpc/codes"
 )

 const (
@@ -51,7 +52,7 @@ type server struct{}

 // SayHello implements helloworld.GreeterServer
 func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
-       return &pb.HelloReply{Message: "Hello " + in.Name}, nil
+       return nil, grpc.Errorf(codes.DataLoss, "too grumpy to greet")
 }

 func main() {
$ go run greeter_server/main.go &
[1] 12775
$ go run greeter_client/main.go 
2015/03/03 13:53:30 could not greet: rpc error: code = 2 desc = "rpc error: code = 15 desc = \"too grumpy to greet\""
exit status 1

I expected to see code=15, now I see code=2 wrapping a code=15.

My reading of the source (https://github.com/grpc/grpc-go/blob/master/server.go#L246 , https://github.com/grpc/grpc-go/blob/master/rpc_util.go#L227 ) says convertCode should have if err, ok := err.(rpcError); ok { return err.code }.

That still leaves the textual double-wrapping; it seems that should not be done on the server-side for rpcError, just send rpcError.desc if the error is a grpcError.

Also, could there be a little less textual wrapping? And could those codes get a go stringer on them? Here's what might be ideal:

could not greet: rpc error: DataLoss: too grumpy to greet

and only start doing quoting if the message is "hostile", as in contains \n etc. That would require defining a safe set of runes.

Add values to base context for a server / service

I'd like to add some values to the base context for a server or a service. For instance a db connection or something. Something like:

1. Bootstrap app
2. Open DB connection
3. Create a context with the db connection as value
4. Set is a the base context for a server
5. All RPC requests are able to access the DB connection through the context.

Performance benchmark, profiling and optimization

We can start with a microbenchmark. But eventually we need a benchmark running as different processes.

Support RPC payload compression

deflate, zippy, etc.

Ghost message appear on server

A simple ping pong rpc with increasing counter. Client send ping every n seconds interval, specifying timeout on context.

What happened:

1. Client-server ping pong smoothly.
2. Turn off network connection.
3. Client failed "context deadline exceeded". Client's main exited.
4. Turn on network connection.
5. After sometime, server received ghost ping message with updated counter.

Expected:
Server shouldn't receive message, since client process already exit.

Seems like low level retransmission problem. I'm not familiar with HTTP2 protocol.

Context not passed from client to server

Hi,

I tried to insert a key-value in the context to client, but didn't get it from server. And I also tried to cancel a context from client, but server didn't get it.

Can you help provide any advice here? Is it gonna be implemented in the future?

Document the usage of context on server

It wasn't clear to me what's the use of context in the generated server interface. In route guide example:

rpc GetFeature(Point) returns (Feature) {}
rpc ListFeatures(Rectangle) returns (stream Feature) {}

Why is GetFeature gets context while ListFeatures dont ?

Bidirectional RPC

Does grpc support bidirection (e.g. server calls client's method anytime)?

Way to access the string description of error from RPC method

Right now, errors seen from rpc calls provide no access to original error string, forcing a prefix:

rpc error: code = 3 desc = "original error goes here"

I would like to display errors from my server to the user, but this wrapping makes that ugly. It's also unlike other Go error handling, where e.g. os.PathError exposes the data items as struct fields, including the underlying error.

Is there a reason this can't be

type RPCError struct {
    Code codes.Code
    Message string
}

with rpc client calls typically returning *RPCError values (pointer to match stdlib behavior).

Also, is there a reason this message looks like program code instead of like all the other errors, e.g. http://golang.org/src/os/error.go?s=600:634#L16 ? That and a String method on Code would make it more pleasant.

https://github.com/golang/oauth2/commit/96e89befdc0f88d5868b297eb0e0727ce6978884 breaks grpc now.

fixing ...

hpack frame decoding fault poisons entire connection

If I have a single gRPC connection and flood it such that a bunch of RPCs end up timing out, the connection gets into an unrecoverable state. Any future RPCs that should otherwise work end up failing with errors like
rpc error: code = 13 desc = "transport: HPACK header decode error: decoding error: invalid indexed representation index 82"
(and the index increments over time).

I traced it to the http2 hpack decoder, which has d.parseHeaderFieldRepr returning an error. Somewhere after that, though, I'd expect the connection to recognise that it is stuck and automatically reset. But it doesn't.

I have a private reproduction that appears to be able to reproduce this 100% of the time. I haven't tried reproducing it with a smaller test case yet.

I don't know who is at fault, grpc-go or http2. But it shouldn't be possible to send a bunch of RPCs on a gRPC connection such that the connection ends up poisoned.

/cc @bradfitz in case he has thoughts.

Ambiguous name: credentials.NewClientTLSFromCert()

My first impression was "TLS client certificate authentication", i.e. distinguish each clients by certificate that they sent. But from cursory look, turns out it's actually certificate pinning.. making sure client talks with pinned server CA.

Am I right, or does grpc actually supports client certificate authentication ?

codes.Internal returned for context.DeadlineExceeded on client

I'm making a client-side RPC call to a grpc-go server that has crashed. My context has a timeout (context.WithTimeout) so after a few seconds the RPC fails as expected with the following message:

2015/03/22 14:01:04 [error] rpc error: code = 13 desc = "stream error: code = 4 desc = \"context deadline exceeded\""

However, the grpc.Code for this error is codes.Internal rather than codes.DeadlineExceeded.

I don't know the codebase well enough to add a failing test case, however I believe the issue comes from this line, which returns all errors with the code codes.Internal:

grpc-go/call.go

Line 150 in fbd3f79

return Errorf(codes.Internal, "%v", err)

credentials package should support oauth2.TokenSource

The google.golang.org/grpc/credentials package should have a function that can be given an oauth2.TokenSource and returns a credentials.Credentials. This might just involve renaming and exporting its existing computeEngine type.

This will make it easier to use "Application Default Credentials" ([1], [2]) with gRPC.

[1] https://developers.google.com/accounts/docs/application-default-credentials
[2] http://godoc.org/golang.org/x/oauth2/google#DefaultTokenSource

README should mention other necessary packages

It'd be useful if the README mentioned running

go get -u github.com/golang/protobuf/{proto,proto-gen-go}

as these packages are otherwise only mentioned in codegen.sh.

Add performance tracing

latency, CPU, etc.

Change gRPC-Go import path

Hi gRPC-Go users,

We just committed a change which changed the import path of grpc from

github.com/grpc/grpc-go/rpc

to

google.golang.org/grpc

Please migrate your code correspondingly. Sorry for the inconvenience.

Kill stream client caused server quit silently

@iamqizhao,
Client send request message to server per second, and server return the responses back, two side are stream mode. to reproduce this issue, I comment out the recv response message from server on client side, keep it several seconds, then killed client, server quit silently almost at the same time, no core dump file found on server side. I did this just want to know how server handle this scenario that client crashed(no deadline or timeout set in context on client side, and CloseSend() will not be called due to client crashed)? will the goroutine quit or just block on server side? thanks.

Context cancelation does not propagate from the client to the server for streams

I have a stream defined that will send messages perpetually to the client, until the client is no longer interested. Since there is no "stream.Close" on the client side (only stream.CloseSend), I assume using context.WithCancel on the client side is the correct way to do this. However, it seems like cancelation is not being propagated to the server: The server continues calling Send (with no errors) long after the client stops caring, and eventually the Send blocks entirely and the server's goroutine hangs. I think the goroutines are unblocked and exit if the connection is closed, but I didn't test that thoroughly.

Unary RPCs seem to honor context cancelation.

Is there some other way for the client to indicate it is no longer interested in the stream?

Support Unix domain socket

Support different data formats besides protobuf

Please test with broken HTTP/2 stream IDs

Hi there,
We recently got hit in C with a crash that happened when HTTP/2 stream IDs went unexpected (violating the invariant of increase on a connection) and also aren't prepared for the situation where we approach or cross max stream id (0x7fffffffu). Please check that this case works in Go. If you need some client code to test that, I can let you use the client code that was sent to me.
This is related to C issues: grpc/grpc#946 and grpc/grpc#957

Thanks!
Vijay

Access to TLS client certificate

I can't see any way for an RPC method to authenticate a client based on a TLS certificate.

An example program where an RPC method echoes the client TLS certificate would be great.

Typo in credentials.TransportAuthenticator.Dial: says "add" should say "addr"

type TransportAuthenticator interface {
    // Dial connects to the given network address and does the authentication
    // handshake specified by the corresponding authentication protocol.
    Dial(add string) (net.Conn, error)

What is encoding of string?

https://github.com/mattn/grpc-example

When passing utf-8 strings as protobuf's string, It can't be decoded as utf-8

Is this problem of grpc-go? or grpc?

optimize to reduce memory alloc and copying?

May I ask what the plan is over here?

grpc-go/rpc_util.go

Line 142 in 9c4157f

// TODO(zhaoq): optimize to reduce memory alloc and copying.

// TODO(zhaoq): optimize to reduce memory alloc and copying.

generated stream handler doesn't include context in method argument

It seems that context is correctly passed to unary methods:

... SayHello(context.Context, *HelloRequest) (*HelloReply, error)

However, for streaming handlers, this is part of the response interface (stream in the example below):

... ListFeatures(rect *pb.Rectangle, stream pb.RouteGuide_ListFeaturesServer) error

This seems slightly inconsistent withe usage guidelines of context.Context. If there is a good reason for this, it would be interesting to understand it. There may be cases where we want to modify the context before sending a stream response:

... ListFeatures(ctx context.Context, rect *pb.Rectangle, stream pb.RouteGuide_ListFeaturesServer) error {
    ctx = WithSpecialContext(ctx, ...)
    return stream.Send(ctx, v)
}

This may be particularly useful if a server has the ability to modify headers and trailers based on the context.

Support graceful server shutdown

A server that is attempting to gracefully shut down a
connection SHOULD send an initial GOAWAY frame with the last stream
identifier set to 2^31-1 and a NO_ERROR code. This signals to the
client that a shutdown is imminent and that no further requests can
be initiated. After waiting at least one round trip time, the server
can send another GOAWAY frame with an updated last stream identifier.
This ensures that a connection can be cleanly shut down without
losing requests.

Please add an example to NewServer

It’s unclear to me whether ServerOptions can be nil.

Also, it’s unclear to me how to initialize an empty ServerOptions, i.e. if I do not want to use MaxConcurrentStreams, what do I pass to NewServer?

An example (embedded into the docs!) would greatly help with this.

interop: go client silently discards service errors

Observed when calling against a service built on the C++ gRPC core.

// Service snippet

class GreeterServiceImpl final : public Greeter::Service {
  Status SayHello(ServerContext* context, const HelloRequest* request,
                  HelloReply* reply) override {
    return Status(grpc::StatusCode::NOT_FOUND);
  }
};

// Client snippet

package main
import (
    "log"
    "os"
    pb "github.com/grpc/grpc-common/go/helloworld"
    "golang.org/x/net/context"
    "google.golang.org/grpc"
)
const (
    address     = "localhost:50051"
    defaultName = "world"
)
func main() {
    // Set up a connection to the server.
    conn, err := grpc.Dial(address)
    if err != nil {
        log.Fatalf("did not connect: %v", err)
    }
    defer conn.Close()
    c := pb.NewGreeterClient(conn)
    // Contact the server and print out its response.
    name := defaultName
    if len(os.Args) > 1 {
        name = os.Args[1]
    }
    r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: name})
    if err != nil {
        log.Fatalf("could not greet: %v", err)
    }
    log.Printf("Greeting: %s", r.Message)
}

Client output:

$ go run greeter_client/*
2015/04/29 18:22:26 Greeting: 

recieved 4294962489-bytes data exceeding the limit 1048560 bytes

I have seen a lot of this exceeding limit error after rev 4320b5b:

2015/04/10 14:35:47 transport: http2Server connection error: desc = "recieved 4294958892-bytes data exceeding the limit 1048560 bytes"

Server and client are running at the same grpc-go version. This happens when I have 1020 clients consume jobs, then streaming updates to server, request speed is about 1001000/s.

What version of Go are you using (go version)?
go version go1.4.2 linux/amd64

What operating system and processor architecture are you using?

Linux instance-1 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Errorf docs refer to CodeOf, mean Code

http://godoc.org/google.golang.org/grpc#Errorf

Errorf returns an error containing an error code and a description; CodeOf extracts the Code. Errorf returns nil if c is OK.