infobloxopen/atlas-cli

atlas-cli's Issues

Reflection support

I added reflection support

❯ grpcurl  -plaintext localhost:9090 list
grpc.reflection.v1alpha.ServerReflection
service.Account

but

❯ grpcurl  -plaintext localhost:9090 list service.Account
Failed to list methods for service "service.Account": Symbol not found: service.Account
caused by: File not found: github.com/lyft/protoc-gen-validate/validate/validate.proto

How do I fix this? Thank you!

Enable engine=postgres in gorm_out

what're your thoughts on enabling engine=postgres in the gorm_out configuration?

New project with atlas-cli does not compile

Repo Source: https://github.com/seizadi/aws-cost

Steps to reproduce:

Install CLI:

> git clone https://github.com/infobloxopen/atlas-cli.git
> cd atlas-cli
> make

Create App:

❯ atlas init-app  -delve -gateway -health -helm -kind -name aws-cost -registry soheileizadi
Generating protobuf files... done!
Starting mod project... done!
Resolving imports... done!
Initializing git repo... done!

Create Compile Error:

❯ make
github.com/seizadi/aws-cost/pkg/pb/service.proto:7:1: warning: Import github.com/envoyproxy/protoc-gen-validate/validate/validate.proto is unused.
github.com/seizadi/aws-cost/pkg/pb/service.proto:8:1: warning: Import protoc-gen-swagger/options/annotations.proto is unused.
....

Compile Error:

#13 10.57 # github.com/seizadi/aws-cost/cmd/server
#13 10.57 cmd/server/main.go:106:38: cannot use "github.com/grpc-ecosystem/grpc-gateway/runtime".WithForwardResponseOption(forwardResponseOption) (type "github.com/grpc-ecosystem/grpc-gateway/runtime".ServeMuxOption) as type "github.com/grpc-ecosystem/grpc-gateway/v2/runtime".ServeMuxOption in argument to gateway.WithGatewayOptions
#13 10.57 cmd/server/main.go:107:38: cannot use "github.com/grpc-ecosystem/grpc-gateway/runtime".WithIncomingHeaderMatcher(gateway.AtlasDefaultHeaderMatcher()) (type "github.com/grpc-ecosystem/grpc-gateway/runtime".ServeMuxOption) as type "github.com/grpc-ecosystem/grpc-gateway/v2/runtime".ServeMuxOption in argument to gateway.WithGatewayOptions
#13 10.57 cmd/server/main.go:110:52: cannot use pb.RegisterAwsCostHandlerFromEndpoint (type func(context.Context, *"github.com/grpc-ecosystem/grpc-gateway/runtime".ServeMux, string, []grpc.DialOption) error) as type gateway.registerFunc in argument to gateway.WithEndpointRegistration

Issue while swagger endpoint serves the swagger json

In swagger handler function, it joins the swagger directory and the suffix ".swagger.json" . The join isn't creating the exact path to the swagger file as the generated swagger file has the name matching with the proto file name(if I am not wrong) and appending ".swagger.json". So do we have to add that name in the swagger directory also?

Default values in config template

We need:

atlas-cli/atlas/config.gotmpl

Lines 69 to 70 in 122ae2e

	defaultKeepaliveTime = 10
	defaultKeepaliveTimeout = 20

and

atlas-cli/atlas/config.gotmpl

Lines 128 to 129 in 122ae2e

	flagKeepaliveTime = pflag.String("config.keepalive.time", defaultKeepaliveTime, "default value, in seconds, of the keepalive time")
	flagKeepaliveTimeout = pflag.String("config.keepalive.timeout", defaultKeepaliveTimeout, "default value, in seconds, of the keepalive timeout")

defined in config template

Though README.md indicates that we need to copy config.go for usage but I see active development in https://github.com/infobloxopen/atlas-cli/blob/master/atlas/templates/cmd/server/config.go.gotmpl

go_package generation

in service.proto, option go_package = "user/sample-app/pkg/pb;pb" is generated
How do I configure it to generate option go_package = "github.com/user/sample-app/pkg/pb;pb"

requestid middleware must be after the grpc_logrus interceptor in the middleware chain

atlas-cli/atlas/templates/cmd/server/grpc.go.gotmpl

Line 20 in d39287c

requestid.UnaryServerInterceptor(),

Support creation outside of `$GOPATH/src`

As it stands, atlas-cli doesn't create projects outside of the $GOPATH/src directory.
Since Go modules makes it optional for projects to be present in $GOPATH, I feel atlas-cli should support creating projects in any directory.

This is the error it throws:

Unable to initialize application: formatting error: project must be initialized inside $GOPATH/src directory or below

Incompatible runtime package and atlas-generated code

This looks like a compatibility issue with the code that is generated by atlas toolkit and the github.com/grpc-ecosystem/grpc-gateway/runtime package.

There are two errors when running the command " go run cmd/server/*.go "
(Ubuntu 16.04)

1. runtime.CamelCaseFieldMask() this function does not exist vendor/github.com/grpc-ecosystem/grpc-gateway/runtime/fieldmask.go

2. runtime.FieldMaskFromRequestBody - this funciton requires an additional argument.

There are many instances of these sets of errors in pkg/pb/service.pb.gw.go (in the generated app)

~/go/src/atid2$ go run cmd/server/*.go
pkg/pb/service.pb.gw.go:148:3: undefined: runtime.CamelCaseFieldMask
pkg/pb/service.pb.gw.go:150:56: not enough arguments in call to runtime.FieldMaskFromRequestBody
        have (io.Reader)
        want (io.Reader, *descriptorpb.DescriptorProto)

How to reproduce

• create the app
• run the app (no customization)
• errors are seen

pj@bay:~/go/src$ atlas init-app --name=atapp9 --registry=prakarp --expand=expand.txt --db=true --helm=true --pubsub=true --gateway=true
Generating protobuf files... done!
Starting mod project... done!
Resolving imports... done!
Initializing git repo... done!
pj@bay:~/go/src$ cd atapp9
pj@bay:~/go/src/atapp9$ go run cmd/server/*.go

Expected Output

(Assuming that config.yaml has a GRPC port configured as 9099 because 9090 is used by systemd on Linux. This is from an app that has been "fixed" - see the comment below that indicates a workaround of using the 'generator' package, from bertha.cloud site).

{"level":"debug","msg":"serving internal http at \"0.0.0.0:8081\"","time":"2020-06-09T00:20:50-07:00"}
{"level":"info","msg":"serving gRPC at 0.0.0.0:9099","time":"2020-06-09T00:20:50-07:00"}
{"level":"info","msg":"serving http at 0.0.0.0:8080","time":"2020-06-09T00:20:50-07:00"}

Actual Output

pj@bay:~/go/src$ atlas init-app --name=atapp9 --registry=prakarp --expand=expand.txt --db=true --helm=true --pubsub=true --gateway=true

pj@bay:~/go/src/atapp9$ go run cmd/server/*.go
#atapp9/pkg/pb
pkg/pb/service.pb.gw.go:175:3: undefined: runtime.CamelCaseFieldMask
pkg/pb/service.pb.gw.go:177:56: not enough arguments in call to runtime.FieldMaskFromRequestBody
        have (io.Reader)
        want (io.Reader, *descriptorpb.DescriptorProto)
pkg/pb/service.pb.gw.go:389:3: undefined: runtime.CamelCaseFieldMask
pkg/pb/service.pb.gw.go:391:56: not enough arguments in call to runtime.FieldMaskFromRequestBody
        have (io.Reader)
        want (io.Reader, *descriptorpb.DescriptorProto)
pkg/pb/service.pb.gw.go:603:3: undefined: runtime.CamelCaseFieldMask
pkg/pb/service.pb.gw.go:605:56: not enough arguments in call to runtime.FieldMaskFromRequestBody
        have (io.Reader)
        want (io.Reader, *descriptorpb.DescriptorProto)
pkg/pb/service.pb.gw.go:817:3: undefined: runtime.CamelCaseFieldMask
pkg/pb/service.pb.gw.go:819:56: not enough arguments in call to runtime.FieldMaskFromRequestBody
        have (io.Reader)
        want (io.Reader, *descriptorpb.DescriptorProto)
pkg/pb/service.pb.gw.go:1031:3: undefined: runtime.CamelCaseFieldMask
pkg/pb/service.pb.gw.go:1033:56: not enough arguments in call to runtime.FieldMaskFromRequestBody
        have (io.Reader)
        want (io.Reader, *descriptorpb.DescriptorProto)
pkg/pb/service.pb.gw.go:1033:56: too many errors
pj@bay:~/go/src/atapp9$

Apply a patch (manually)

Patch is available from: https://gitlab.bertha.cloud/partitio/lab/micro-gateway/-/blob/atlas-patch/runtime/fieldmask.go

(Edit the file fieldmask.go.... I have a copy of the file fieldmask.go that has been edited and simply copy over to the generated app like below.

pj@bay:~/go/src/atapp9$ cp ~/fieldmask.go vendor/github.com/grpc-ecosystem/grpc-gateway/runtime
/fieldmask.go

Run the App again

This time the app runs successfully.. the configuration values for the Postgres database are wrong and so the app exits.. That is not related to the issue.

pj@bay:~/go/src/atapp9$ go run cmd/server/*.go
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
        A future release of golang/protobuf will delete this package,
        which has long been excluded from the compatibility promise.

2020/06/09 12:31:47 Serving from default values, environment variables, and/or flags
{"level":"info","msg":"pubsub: subscribing to server at pubsub.atlas:5555 with topic \"example_hello\" and subscription ID \"example_hello_subscriberid\"","time":"2020-06-09T12:31:47-07:00"}
{"level":"debug","msg":"serving internal http at \"0.0.0.0:8081\"","time":"2020-06-09T12:31:47-07:00"}
{"level":"fatal","msg":"pq: password authentication failed for user \"postgres\"","time":"2020-06-09T12:31:47-07:00"}
exit status 1
pj@bay:~/go/src/atapp9$

Ubuntu 16.04

remove period delimiters from project

Periods having meaning to many parsers. JSON and YAML for instance will assign different types to app-name vs app.name. TOML requires brackets to support periods. This is why you rarely, if ever, see periods used to delimit strings in any language. We have introduced periods to nearly every aspect of this project. As a result we are forced to do strange things to support something that has dubious technical reasons for its introduction.

We will remove periods from the project and replace them with the preferred delimiter of the code being generated.

Helm camelCase https://helm.sh/docs/chart_best_practices/values/#naming-conventions

chicken: true
chickenNoodleSoup: true

Viper is case insensitive, use stapler case for cli args spf13/viper#635. They use crazy case to showcase the insensitivity. - is a safer choice to avoid conflicting keys.

logrus-level: info

yaml camelCase - follow the conventions found in k8s itself

apiVersion: apps/v1

json camelCase - following k8s practice

{
  "apiVersion": "apps/v1"
}

Please use latest golang builder image

Current version is 1.13.1.
https://github.com/infobloxopen/atlas-cli/blob/master/atlas/templates/docker/Dockerfile.gotmpl

Incorrect import path

in service.poroto,
import "google/api/annotations.proto"; was generated

But import _ "google.golang.org/genproto/googleapis/api/annotations" should be generated instead.

atlas init-app - "Starting mod project exit status 1"

~/go/src$ atlas init-app -name=atest2 -gateway -db -registry=infoblox -pubsub -health
Generating protobuf files... done!
Starting mod project... exit status 1

Is the above an error? if so any tips on resolving it?
go version go1.14.2 linux/amd64

Had done the following in the previous steps, but might not be consequential:

~/go/src/github.com/infobloxopen/atlas-cli$ make
main.go:1: running "go-bindata": exec: "go-bindata": executable file not found in $PATH
Makefile:12: recipe for target 'templating' failed
make: *** [templating] Error 1

Then installed go-bindata tool with:

$ go get github.com/jteeuwen/go-bindata 

~/go/src/github.com/jteeuwen/go-bindata$ go build
~/go/src/github.com/jteeuwen/go-bindata$
~/go/src/github.com/infobloxopen/atlas-cli$ export PATH=$PATH:~/go/bin
~/go/src/github.com/infobloxopen/atlas-cli$ make
atlas/templates/template-bindata.go
~/go/src/github.com/infobloxopen/atlas-cli$ which atlas
/home/.../go/bin/atlas

Deployment fails when database is enabled

When I tried to generate and deploy a service with -db option in kind environment got this error

make kind-deploy:

Switched to context "kind-minikube".
dapr is already installed
Error from server (AlreadyExists): namespaces "murad0660" already exists
No resources found.
redis is already installed
component.dapr.io/dapr-component created
bin/helm package helm/storage-infoblox-test --version v0.0.1-0-gdea63ed-unreleased
Successfully packaged chart and saved it to: /home/murad/go/src/github.com/smailic05/storage-infoblox-test/storage-infoblox-test-v0.0.1-0-gdea63ed-unreleased.tgz
Image: "murad0660/storage-infoblox-test:v0.0.1-0-gdea63ed-unreleased" with ID "sha256:4c5ea1529f2af707d157b7bf3cdac5ce0b18037ee84a0d8998f9dc43c57d76c8" not yet present on node "minikube-control-plane", loading...
Image: "murad0660/storage-infoblox-test:v0.0.1-0-gdea63ed-unreleased" with ID "sha256:4c5ea1529f2af707d157b7bf3cdac5ce0b18037ee84a0d8998f9dc43c57d76c8" not yet present on node "minikube-worker", loading...
Image: "murad0660/storage-infoblox-test:v0.0.1-0-gdea63ed-unreleased" with ID "sha256:4c5ea1529f2af707d157b7bf3cdac5ce0b18037ee84a0d8998f9dc43c57d76c8" not yet present on node "minikube-worker2", loading...
make[1]: Entering directory '/home/murad/go/src/github.com/smailic05/storage-infoblox-test'
warning: GOPATH set to GOROOT (/home/murad/go) has no effect
Error from server (AlreadyExists): namespaces "murad0660" already exists
bin/helm is already downloaded, version.BuildInfo{Version:"v3.5.3", GitCommit:"041ce5a2c17a58be0fcd5f5e16fb3e7e95fea622", GitTreeState:"dirty", GoVersion:"go1.15.8"}
serviceaccount/storage-infoblox-test-serviceaccount created
secret/storage-infoblox-test-sakey created
secret/storage-infoblox-test-db-key created
service/storage-infoblox-test-postgres created
service/storage-infoblox-test created
pod/storage-infoblox-test-migration created
error: error validating "deploy-storage-infoblox-test.yaml": error validating data: [ValidationError(Deployment.spec.template.spec.containers[0].env[1].valueFrom): unknown field "key" in io.k8s.api.core.v1.EnvVarSource, ValidationError(Deployment.spec.template.spec.containers[0].env[1].valueFrom): unknown field "name" in io.k8s.api.core.v1.EnvVarSource]; if you choose to ignore these errors, turn validation off with --validate=false
make[1]: *** [Makefile.kind:67: deploy-storage-infoblox-test] Error 1
make[1]: Leaving directory '/home/murad/go/src/github.com/smailic05/storage-infoblox-test'
make: *** [Makefile.kind:76: kind-deploy-storage-infoblox-test] Error 2

atlas init-app throws error "Starting mod project... exit status 1"

As described in the readme I pulled atlas-cli binary and ran atlas init-app -name=heka.cloud. But it throws below error.

sk@sk-VirtualBox:~/go/src/github.com/Infoblox-CTO$ atlas init-app -name=heka.cloud
Generating protobuf files... done!
Starting mod project... exit status 1

But i could see that project folder got created with name heka.cloud. Then i created go.mod and ran go mod vendor command which also throws error for protoc-gen-validate.

sk@sk-VirtualBox:~/go/src/github.com/Infoblox-CTO/heka.cloud$ cat go.mod 
module github.com/Infoblox-CTO/heka.cloud
go 1.14

sk@sk-VirtualBox:~/go/src/github.com/Infoblox-CTO/heka.cloud$ go mod vendor
go: finding module for package github.com/prometheus/client_golang/prometheus/promhttp
go: finding module for package google.golang.org/grpc
go: finding module for package github.com/grpc-ecosystem/go-grpc-prometheus
go: finding module for package github.com/infobloxopen/atlas-app-toolkit/server
go: finding module for package github.com/sirupsen/logrus
go: finding module for package github.com/infobloxopen/atlas-app-toolkit/gorm/resource
go: finding module for package github.com/spf13/pflag
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware/validator
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware
go: finding module for package github.com/infobloxopen/atlas-app-toolkit/requestid
go: finding module for package github.com/infobloxopen/atlas-app-toolkit/gateway
go: finding module for package github.com/grpc-ecosystem/go-grpc-middleware/logging/logrus
go: finding module for package github.com/golang/protobuf/proto
go: finding module for package github.com/golang/protobuf/ptypes
go: finding module for package google.golang.org/genproto/googleapis/api/annotations
go: finding module for package github.com/golang/protobuf/ptypes/empty
go: finding module for package github.com/spf13/viper
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options
go: finding module for package github.com/lyft/protoc-gen-validate/validate
go: found github.com/grpc-ecosystem/go-grpc-middleware in github.com/grpc-ecosystem/go-grpc-middleware v1.2.0
go: found github.com/grpc-ecosystem/go-grpc-prometheus in github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: found github.com/infobloxopen/atlas-app-toolkit/gateway in github.com/infobloxopen/atlas-app-toolkit v0.22.0
go: found github.com/prometheus/client_golang/prometheus/promhttp in github.com/prometheus/client_golang v1.7.1
go: found github.com/sirupsen/logrus in github.com/sirupsen/logrus v1.6.0
go: found github.com/spf13/pflag in github.com/spf13/pflag v1.0.5
go: found github.com/spf13/viper in github.com/spf13/viper v1.7.0
go: found google.golang.org/grpc in google.golang.org/grpc v1.30.0
go: found github.com/golang/protobuf/proto in github.com/golang/protobuf v1.4.2
go: found github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger/options in github.com/grpc-ecosystem/grpc-gateway v1.14.6
go: found github.com/lyft/protoc-gen-validate/validate in github.com/lyft/protoc-gen-validate v0.4.0
go: found google.golang.org/genproto/googleapis/api/annotations in google.golang.org/genproto v0.0.0-20200711021454-869866162049
go: github.com/Infoblox-CTO/heka.cloud/pkg/pb imports
	github.com/lyft/protoc-gen-validate/validate: github.com/lyft/[email protected]: parsing go.mod:
	module declares its path as: github.com/envoyproxy/protoc-gen-validate
	        but was required as: github.com/lyft/protoc-gen-validate

I even set GO111MODULE="auto" but no luck.

sk@sk-VirtualBox:~/go/src/github.com/Infoblox-CTO$ go env
GO111MODULE="auto"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/sk/.cache/go-build"
GOENV="/home/sk/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/sk/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build284363722=/tmp/go-build -gno-record-gcc-switches"

How to resolve this exit status 1 and go mod vendor error?

Code generated with Expand option does not work

Although it compiles the Endpoints are not registered properly and some migrations files are missing. There is a
writeup of all the issues here,
it is not intended to help people fix the problems manually but identify all the gaps in the autogenerated code.

Feature request: service scaffold

Similar to grapi g scaffold-service books from https://github.com/izumin5210/grapi

change defaultInternalHealth from /health to /healthz as its referenced in deployment file to make it consistent

Ref: https://github.com/infobloxopen/atlas-cli/blob/master/atlas/templates/deploy/config.yaml.gotmpl#L45

variable collision in makefile.common and makefile.vars

DOCKER_RUNNER is defined in makefile.vars and makefile.common

This causes make targets like fmt and test to fail because the PWD is set to a directory which doesn't exist in the container

Support go modules instead of dep for package management

Would be great to have go modules OOB instead of dep.
Please be aware that, @kd7lxl and I found out that in some scenarios, we are seeing

[atlas.log.collector] Running shell script
+ make fmt
go: github.com/Infoblox-CTO/[email protected]: invalid version: git fetch -f origin refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/ubuntu/workspace/TO_atlas.log.collector_PR-4-5G2BA45OWJUUZLL6DTOAOYBTN7W45TQJ4SEECAOTP7V6MTYQWR3A/pkg/mod/cache/vcs/9a3fcdf7744bc56b2631e58484cb8416d1983033bab18fd28f53ac1feaf1d09d: exit status 128:
    fatal: could not read Username for 'https://github.com': terminal prompts disabled
go: github.com/Infoblox-CTO/[email protected]: invalid version: git fetch -f https://github.com/Infoblox-CTO/atlas.ophidauth.middleware refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* in /home/ubuntu/workspace/TO_atlas.log.collector_PR-4-5G2BA45OWJUUZLL6DTOAOYBTN7W45TQJ4SEECAOTP7V6MTYQWR3A/pkg/mod/cache/vcs/9a3fcdf7744bc56b2631e58484cb8416d1983033bab18fd28f53ac1feaf1d09d: exit status 128:
    fatal: could not read Username for 'https://github.com': terminal prompts disabled
Makefile.common:50: recipe for target 'fmt-atlas' failed
make: *** [fmt-atlas] Error 1

Tom has this PR to temporary workaround
https://github.com/Infoblox-CTO/atlas.log.collector/pull/6#partial-pull-merging

unusable project created by default

Reproduction Steps:

atlas init-app -db -name acme -pubsub

Project is created. But does not compile.

# github.com/infobloxopen/acme/cmd/server
./grpc.go:40:12: undefined: svc
./grpc.go:44:2: undefined: pb
./main.go:5:2: imported and not used: "flag"
./main.go:9:2: imported and not used: "github.com/infobloxopen/acme/vendor/github.com/grpc-ecosystem/go-grpc-middleware" as grpc_middleware
./main.go:10:2: imported and not used: "github.com/infobloxopen/acme/vendor/github.com/grpc-ecosystem/go-grpc-middleware/validator" as grpc_validator
./main.go:81:34: undefined: fmt
./main.go:96:26: undefined: strings
./main.go:99:3: undefined: log
./main.go:102:4: undefined: log
./main.go:105:3: undefined: log
./main.go:105:3: too many errors```