izumin5210/grapi

😮 A surprisingly easy API server and generator in gRPC and Go

😮 A surprisingly easy API server and generator in gRPC and Go

Features

• You can develop and deploy API servers blazingly fast ⚡
• Easy code generator
  • application (inspired by rails new and create-react-app)
  • gRPC services and their implementations (inspired by rails g (scaffold_)controller)
• User-friendly protoc wrapper (inspired by protoeasy)
• Provides gRPC and HTTP JSON API with single implementation by using grpc-gateway
• Generates codes based on google's API design guideline

⚠️ Migrate 0.4.x -> 0.5.x ⚠️

grapiserver will not handle os signals from v0.5.x. We recommend to use appctx.Global() if you want to handle them.

📝 How to migrate

0. Bump grapi version
   • `go get -u github.com/izumin5210/[email protected]'
1. Update cmd/server/run.go

   •  	// Application context
     -	ctx := context.Background()
     +	ctx := appctx.Global()

   • -	return s.ServeContext(ctx)
     +	return s.Serve(ctx)

⚠️ Migrate 0.3.x -> 0.4.x ⚠️

Some tools that are depended by grapi are updated. If you have a grapi project <=v0.3.x, you should migrate it.

📝 How to migrate

0. Bump grapi version
   • If you use dep, update Gopkg.toml

      [[constraint]]
        name = "github.com/izumin5210/grapi"
     -  version = "0.3.0"
     +  version = "0.4.0"

   • and run dep ensure
1. Update gex and tools.go

   • go get -u github.com/izumin5210/gex/cmd/gex
     gex --regen
     

2. Initialize Go Modules

   • go mod init
     go mod tidy
     

3. Update grapi.toml

   • package = "yourcompany.yourappname"
     
     [grapi]
     server_dir = "./app/server"
     
     [protoc]
     protos_dir = "./api/protos"
     out_dir = "./api"
     import_dirs = [
       "./api/protos",
     -  "./vendor/github.com/grpc-ecosystem/grpc-gateway",
     -  "./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis",
     +  '{{ module "github.com/grpc-ecosystem/grpc-gateway" }}',
     +  '{{ module "github.com/grpc-ecosystem/grpc-gateway" }}/third_party/googleapis',
     ]
     
       [[protoc.plugins]]
       name = "go"
       args = { plugins = "grpc", paths = "source_relative" }
     
       [[protoc.plugins]]
       name = "grpc-gateway"
       args = { logtostderr = true, paths = "source_relative" }
     
       [[protoc.plugins]]
       name = "swagger"
       args = { logtostderr = true }

4. Drop dep

   • rm Gopkg.*
     

⚠️ Migrate 0.2.x -> 0.3.x ⚠️

grapi v0.3.0 has some breaking changes. If you have a grapi project <=v0.2.x, you should migrate it.

📝 How to migrate

0. Bump grapi version
   • If you use dep, update Gopkg.toml

      [[constraint]]
        name = "github.com/izumin5210/grapi"
     -  version = "0.2.2"
     +  version = "0.3.0"

   • and run dep ensure
1. Introduce gex

   • go get github.com/izumin5210/gex/cmd/gex
     

2. Add defualt generator plugins:

   • gex \
       --add github.com/izumin5210/grapi/cmd/grapi \
       --add github.com/izumin5210/grapi/cmd/grapi-gen-command \
       --add github.com/izumin5210/grapi/cmd/grapi-gen-service \
       --add github.com/izumin5210/grapi/cmd/grapi-gen-scaffold-service \
       --add github.com/izumin5210/grapi/cmd/grapi-gen-type
     

3. Add protoc plugins via gex

   • gex \
       --add github.com/golang/protobuf/protoc-gen-go \
       --add github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway \
       --add github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger
     

   • Remove protoc plugins from Gopkg.toml

     -required = [
     -  "github.com/golang/protobuf/protoc-gen-go",
     -  "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway",
     -  "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger",
     -]

4. Update grapi.toml

   • +package = "yourcompany.yourappname"
     +
      [grapi]
      server_dir = "./app/server"
     
      [protoc]
      protos_dir = "./api/protos"
      out_dir = "./api"
      import_dirs = [
     +  "./api/protos",
        "./vendor/github.com/grpc-ecosystem/grpc-gateway",
        "./vendor/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis",
      ]
     
        [[protoc.plugins]]
     -  path = "./vendor/github.com/golang/protobuf/protoc-gen-go"
        name = "go"
        args = { plugins = "grpc", paths = "source_relative" }
     
        [[protoc.plugins]]
     -  path = "./vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway"
        name = "grpc-gateway"
     -  args = { logtostderr = true }
     +  args = { logtostderr = true, paths = "source_relative" }
     
        [[protoc.plugins]]
     -  path = "./vendor/github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger"
        name = "swagger"
        args = { logtostderr = true }

Getting Started

Create a new application

$ grapi init awesome-app

Create a new service

$ grapi g service books

Or, if you need full standard methods, you can get them with following command:

$ grapi g scaffold-service books

And you should register generated services to the grapiserver.Engine instance:

 // app/run.go
 
 // Run starts the grapiserver.
 func Run() error {
 	s := grapiserver.New(
 		grapiserver.WithDefaultLogger(),
 		grapiserver.WithServers(
+			server.NewBookServiceServer(),
-		// TODO
 		),
 	)
 	return s.Serve()
 }

If you updated service definition, you can re-generate .pb.go and .pb.gw.go with following command:

$ grapi protoc

Start server

$ grapi server

User-defined commands

$ grapi g command import-books
$ vim cmd/import-books/run.go  # implements the command
$ grapi import-books  # run the command

Build commands (including server)

$ grapi build

Installation

1. grapi
   • Linux
     • curl -Lo grapi https://github.com/izumin5210/grapi/releases/download/v0.2.2/grapi_linux_amd64 && chmod +x grapi && sudo mv grapi /usr/local/bin
   • macOS
     • brew install izumin5210/tools/grapi
   • others
     • go get github.com/izumin5210/grapi/cmd/grapi
2. dep or Modules
   • dep
     • macOS
       • brew install dep
     • others
       • See Installation · dep
       • curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
   • Modules (experimental)
     • Use Go 1.11 and set GO111MODULE=on your env vars
3. protoc
   • macOS
     • brew install protobuf
   • others
     • Download and install from google/protobuf

grapi's Issues

window grapi Filename too long

grouped write of manifest, lock and vendor: error while writing out vendor tree:
failed to write dep tree: failed to export github.com/izumin5210/grapi: error:
unable to create file C:\Users\ADMINI~1\AppData\Local\Temp\dep227690330\vendor\g
ithub.com\izumin5210\grapi\pkg/grapicmd/internal/module/generator/.snapshots/Tes
t_ServiceGenerator-foo-bar-baz_with_list,create,rename,delete,move_move-Generate
-app-server-foo-bar_baz_server_register_funcs.go: Filename too long
: exit status 128
failed to execute dep ensure: exit status 1��

proposal: gen the DB

THis is a really great project. I have been working on something similar, but i think i will swap to yours.

I was wondering if you think using sqlboiler ( github.com/volatiletech/sqlboiler ) to gen the DB makes sense ?
I noticed that you gen the Standard CRUD Methods and so it seems a pretty good idea to then do the same for the DB.

There is always a imbalance, for want of a better word, between what your DB model looks like and what your GRPC model looks like. Hope you know what i mean ?
SO for now, if you think this is a good idea, then the developer has to write code in the middle to get the data from the DB that he / she needs to fulfil the Protobuf API.

In the past i wrote a very large code generator that is quite a bit like this.
In order to help with the "imbalance" we created a mapping between the protobuf and the db tables, to help do more codegen. But i think thats much later..

anyways, eager to hear what your think.

unknown field 'Execer' in struct literal of type gex.Config failed to build github.com/izumin5210/grapi/cmd/grapi: exit status 2

I can't use grapi

❯ cat go.mod
module github.com/rerost/XXX

go 1.13

require (
        github.com/izumin5210/gex v0.6.0
        github.com/izumin5210/grapi v0.5.0
)

❯ gex grapi gen
# github.com/izumin5210/grapi/pkg/protoc
../../../../pkg/mod/github.com/izumin5210/[email protected]/pkg/protoc/providers.go:40:4: unknown field 'Execer' in struct literal of type gex.Config
failed to build github.com/izumin5210/grapi/cmd/grapi: exit status 2

❯ gex grapi
# github.com/izumin5210/grapi/pkg/protoc
../../../../pkg/mod/github.com/izumin5210/[email protected]/pkg/protoc/providers.go:40:4: unknown field 'Execer' in struct literal of type gex.Config
failed to build github.com/izumin5210/grapi/cmd/grapi: exit status 2

Gzip compression

@izumin5210 Is it possible to enable gzip compression for the response specifically for HTTP endpoint.? What would the options look like when bootstrapping the server?

`grapi g type NAME` subcommand

Dependabot can't parse your go.mod

Dependabot couldn't parse the go.mod found at /go.mod.

The error Dependabot encountered was:

go: golang.org/x/[email protected] requires
	golang.org/x/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /opt/go/gopath/pkg/mod/cache/vcs/e8cbb9fcbcd0d26d8c9a3b58f621a8129147a3b347856e391df04e0a9554a4ed: exit status 128:
	fatal: unable to access 'https://go.googlesource.com/xerrors/': The requested URL returned error: 502

View the update logs.

Fix order of options in proto files that will be generated

https://cloud.google.com/apis/design/file_structure#proto_options

"Connection reset by peer" when sending PATCH request over HTTP if GRPC & Gateway are on same addr

When I have an endpoint that uses the PATCH method, I can't send HTTP requests to that endpoint

$ curl -vvv -X PATCH http://127.0.0.1:3001/api/tasks/4 -d '{...}'
*   Trying 127.0.0.1:3001...
* Connected to 127.0.0.1 (127.0.0.1) port 3001 (#0)
> PATCH /api/tasks/4 HTTP/1.1
> Host: 127.0.0.1:3001
> User-Agent: curl/7.79.1
> Accept: */*
> Content-Length: 72
> Content-Type: application/x-www-form-urlencoded
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

Other HTTP methods work fine.

This appears to be because grapiserver uses cmux to multiplex GRPC and HTTP on the same address:

grapi/pkg/grapiserver/cmux.go

Line 36 in cf43ca2

return s.mux.Match(cmux.HTTP2(), cmux.HTTP1Fast())

... and uses cmux's HTTP1Fast() matcher, which does not recognize the PATCH method:

https://github.com/soheilhy/cmux/blob/v0.1.5/matchers.go#L46-L64

With employer approval, I'll send a pull request to pass "PATCH" to the call to HTTP1Fast() in grapiserver/cmux.go.

Workaround: when I use a different addr for GRPC and HTTP, everything works fine.

Add .gitignore to template

WHY

When I tried to use grapi to start a new project, lots of binary generated in bin directory, which I don't want to include int the initial commit

WHAT

Include .gitignore to the init template to make this a bit easier.

Change directory structure

WHY

to follow golang-standards/project-layout style

WHAT

before	after
. ├── Gopkg.lock ├── Gopkg.toml ├── api │   ├── protos │   │   ├── type │   │   └── *.proto │   ├── type │   ├── *.pb.go │   ├── *.pb.gw.go │   └── *.swagger.json ├── app │   ├── run.go │   └── server │   ├── *_server.go │   ├── *_server_register_funcs.go │   └── *_server_test.go ├── bin ├── cmd │   └── server │   └── run.go ├── grapi.toml ├── tmp └── tools.go	. ├── Gopkg.lock ├── Gopkg.toml ├── api │   ├── protos │   │   ├── type │   │   └── *.proto │   ├── type │   ├── *.pb.go │   ├── *.pb.gw.go │   └── *.swagger.json ├── pkg │   └── server │      ├── cmd │   │ └── cmd.go │   ├── *_server.go │   ├── *_server_register_funcs.go │   └── *_server_test.go ├── bin ├── cmd │   └── server │   └── main.go ├── grapi.toml ├── tmp └── tools.go

failed to initialize project: failed to get the import path

Using Ubuntu 19.04 and the provided install method I am not able to init a project:

failed to initialize project: failed to get the import path

Execute `gofmt` and `goimports` after generating .go files

"Getting started" documentation is incomplete

My apologies if there's some more complete documentation somewhere else that I can't find.

The documentation in README is not complete. That is: it doesn't tell you everything you need to do to get an example application running. In particular, I suspect that I'm supposed to create my gRPC proto service definition somewhere for grapi to process. But I don't see that documented anywhere.

What would be helpful would be a complete walkthrough with step by step instructions of how to produce an example application. Either that, or a bit more of an explanation of the operating principle so that I can at least derive my own instructions.

failed to initialise project

failed to initialise project. I using ubuntu 20.04, go 1.15

grapi init awesome-app
  ➜  Initialize project
failed to initialize project: failed to get the import path

Support Go Modules

• Generates go.mod on init new apps
  • Should work with --HEAD option
• Builds plugins with grapi protoc command

Deployment

What's the recommended way to deploy a grapi project with TLS?
Do I place it behind envoy?

Failed to extract ServerMetadata from context

I've created a new project and generated 1 service.
when I run grpcurl -plaintext 0.0.0.0:3000 list, the server shows Failed to extract ServerMetadata from context

grapi 0.3.2

`grapi protoc` has not display error messages

How to specify a timeout?

I would like to specify a timeout to my RPC calls. Any idea what config needs to be modified and where?

Dependabot couldn't find a Gopkg.toml for this project

Dependabot couldn't find a Gopkg.toml for this project.

Dependabot requires a Gopkg.toml to evaluate your project's current Go dependencies. It had expected to find one at the path: /Gopkg.toml.

If this isn't a Go project, or if it is a library, you may wish to disable updates for it from within Dependabot.

You can mention @dependabot in the comments below to contact the Dependabot team.

Fix go_package options to support nested packages

-option go_package = "api_pb"
+option go_package = "github.com/izumin5210/testapp/api;api_pb"

Print registered services

• subcommand such as rails routes
• log routes on start server such as gin-gonic/gin

CORS

How do I add CORS?

Migration guide for 0.3.x 0.4.x

How do I migrate an app from 0.3.x to 0.4.x?

Extending grapi protoc

For the command grapi protoc, is there a way to add custom plugins such as lyft/protoc-gen-validate

Command line options to modify server addrs

• --grpc-port
• --grpc-host
• --gateway-port
• --gateway-host

Add field mask

What are your thoughts on adding google.protobuf.FieldMask update_mask = 2; to the standard methods generator?

Limit on Concurrent HTTP connections?

Is there a limit on the number of concurrent HTTP connections?. I see that the requests get timed out if there are more than 1000 connections. Is this a config set somewhere which we can tweak?

Grapi init does not work with non-example application names

When I run:

grapi init awesome-app

Grapi initializes properly:

➜ grapi init awesome-app
  ➜  Initialize project
     ╌  cmd/server/main.go
     ╌  cmd/server/run.go
     ╌  grapi.toml
     ╌  tools.go

  ▸  Install dependencies
go: downloading github.com/google/go-cmp v0.5.0
go: downloading github.com/bradleyjkemp/cupaloy/v2 v2.5.0
go: downloading github.com/golang/mock v1.2.0
go: downloading github.com/stretchr/testify v1.5.1
go: downloading github.com/onsi/ginkgo v1.6.0
go: downloading github.com/onsi/gomega v1.4.2
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
go: downloading github.com/BurntSushi/toml v0.3.1
go: downloading golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543
go: downloading github.com/kr/pretty v0.1.0
go: downloading github.com/hpcloud/tail v1.0.0
go: downloading gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7
go: downloading gopkg.in/fsnotify.v1 v1.4.7
go: downloading github.com/kr/text v0.1.0
go: finding module for package google.golang.org/grpc/examples/helloworld/helloworld
go: downloading google.golang.org/grpc/examples v0.0.0-20210326170912-4a19753e9dfd
go: found google.golang.org/grpc/examples/helloworld/helloworld in google.golang.org/grpc/examples v0.0.0-20210326170912-4a19753e9dfd
go: downloading google.golang.org/genproto v0.0.0-20200806141610-86f49bd18e98

When I run:

grapi init anything-else-app

I get the following results:

➜ grapi init anything-else-app
  ➜  Initialize project
     ✔  cmd/server/main.go
     ✔  cmd/server/run.go
     ✔  grapi.toml
     ✔  tools.go

  ▸  Install dependencies
go: creating new go.mod: module anything-else-app
go: to add module requirements and sums:
	go mod tidy
go get: added github.com/izumin5210/grapi v0.5.0
go get: added github.com/srvc/appctx v0.1.0
go get: added google.golang.org/grpc v1.36.1
../../../pkg/mod/github.com/grpc-ecosystem/[email protected]/protoc-gen-grpc-gateway/descriptor/grpc_api_configuration.go:9:2: missing go.sum entry for module providing package github.com/ghodss/yaml (imported by github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/descriptor); to add:
	go get github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway/[email protected]
../../../pkg/mod/github.com/grpc-ecosystem/[email protected]/protoc-gen-grpc-gateway/main.go:17:2: missing go.sum entry for module providing package github.com/golang/glog (imported by github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway); to add:
	go get github.com/grpc-ecosystem/grpc-gateway/[email protected]
failed to install dependencies: failed to build github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway: exit status 1

I have also tested at least a dozen other initialization names, all failed.

I expected grapi init anything-else-app to behave as grapi init awesome-app does, but create a different directory.

go version go1.16.2 darwin/amd64
dep version : v0.5.4
OSX: 11.2.3
Grapi installed via brew: grapi v0.5.0 (go1.13.1 darwin/amd64)

Dependabot can't resolve your Go dependency files

Dependabot can't resolve your Go dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

rsc.io/quote/[email protected]: unrecognized import path "rsc.io/quote/v3" (parse https://rsc.io/quote/v3?go-get=1: no go-import meta tags ())

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

View the update logs.

`grapi build server` fails

WHY

I have a repository that contains more than 20 commands, in which sometimes I want to build only one command to save time.
And it looks like grapi offers a way to do that like grapi build <command name> but this doesn't work

$ grapi build server
  ▸  Building server
can't load package: named files must be .go files: server
failed to build [/Users/potsbo/.go/src/github.com/<repo-slug>/cmd/server/main.go /Users/potsbo/.go/src/github.com/<repo-slug>/cmd/server/run.go]: exit status 1

I did a bit of investigation to find out the cause is https://github.com/izumin5210/grapi/blob/master/pkg/grapicmd/internal/module/script/script.go#L67

Since grapi passes the args I give as an arg to the go build command, the go command receives an unexpected arg.

WHAT

I would like to fix this problem by doing below

• not to take args in Build func
• not to add the args into go build args

But before doing that I would like to know your intention of this code.
Maybe you wanted to let users to pass build args to go? But this doesn't work because cobra doesn't let us passing args other than specified ones, which means this cannot be done in the current architecture.

$ grapi build server -a
unknown shorthand flag: 'a' in -a

Any suggestion? @izumin5210

(*grapiserver.Engine).ServeContext(context.Context, net.Listener)

Example to integrate Sentry

Hi @izumin5210 can you give an example of how to integrate and use sentry with the api server?.

I am able to capture errors using the instructions mentioned here:
https://docs.sentry.io/clients/go/

However capturing panics seems to be difficult since if you get a panic in one of the handlers the server crashes and there is no recovery other than restarting the process?

please advise.

permission request

hey @izumin5210 , can you guide me more on how to contribute opensource project.

Options to use gogo, gofast, gofaster, etc. when generate go code from protobuf

API returns a stream stops streaming after 8 seconds when accessed through grpc-gateway using HTTP

I'm stumped by this problem, please help.

I used grapi to create a server skeleton, then implemented an API that returns a stream.

# proto file
rpc GetTopAccounts (GetTopAccountRequest) returns (stream Account) {

# implementation
for {
for _, acc := range accounts {
		err = svr.Send(acc)
		if err != nil {
			log.Printf("error streaming %+v", err)
			return nil
		}
	}

	log.Printf("iteration %d", i)
	time.Sleep(1000 * time.Millisecond)
}

When calling this API with golang gRPC client, things work fine. However, if I use curl to access the same API using HTTP, the response will stop streaming after exactly 8 seconds.

Here is my code repo
To run it:

go run ./cmd/server

#open another window

curl http://localhost:3100/top/10
...
{"result":{"account_id":"123"}}
{"result":{"account_id":"123"}}
{"result":{"account_id":"123"}}
# after exactly 8 seconds
curl: (18) transfer closed with outstanding read data remaining

the server prints this in console
2019/10/09 21:41:11 iteration 19
2019/10/09 21:41:11 error streaming rpc error: code = Canceled desc = context canceled

Meanwhile, running the gRPC client calling the same API can does not exhibit this problem

go run ./cmd/client
...
streams forever
...