stormcat24/protodep

Collect necessary .proto files (Protocol Buffers IDL) and manage dependencies

protodep

Dependency tool for Protocol Buffers IDL file (.proto) vendoring tool.

Motivation

In building Microservices architecture, gRPC with Protocol Buffers is effective. When using gRPC, your application will depend on many remote services.

If you manage proto files in a git repository, what will you do? Most remote services are managed by git and they will be versioned. We need to control which dependency service version that application uses.

Install

go install

$ go install -v github.com/stormcat24/[email protected]

from binary

Support as follows:

• protodep_darwin_amd64.tar.gz
• protodep_darwin_arm64.tar.gz
• protodep_linux_386.tar.gz
• protodep_linux_amd64.tar.gz
• protodep_linux_arm.tar.gz
• protodep_linux_arm64.tar.gz

$ wget https://github.com/stormcat24/protodep/releases/download/0.1.4/protodep_darwin_amd64.tar.gz
$ tar -xf protodep_darwin_amd64.tar.gz
$ mv protodep /usr/local/bin/

Usage

protodep.toml

Proto dependency management is defined in protodep.toml.

proto_outdir = "./proto"

[[dependencies]]
  target = "github.com/stormcat24/protodep/protobuf"
  branch = "master"

[[dependencies]]
  target = "github.com/grpc-ecosystem/grpc-gateway/examples/examplepb"
  revision = "v1.2.2"
  path = "grpc-gateway/examplepb"

# blacklist by "ignores" attribute
[[dependencies]]
  target = "github.com/kubernetes/helm/_proto/hapi"
  branch = "master"
  path = "helm/hapi"
  ignores = ["./release", "./rudder", "./services", "./version"]
  
# whitelist by "includes" attribute
[[dependencies]]
  target = "github.com/protodep/catalog/hierarchy"
  branch = "main"
  includes = [
    "/protodep/hierarchy/service.proto",
    "**/fuga/**",
  ]
  protocol = "https"

protodep up

In same directory, execute this command.

$ protodep up

If succeeded, protodep.lock is generated.

protodep up -f (force update)

Even if protodep.lock exists, you can force update dependenies.

$ protodep up -f

[Attention] Changes from 0.1.0

From protodep 0.1.0 supports ssh-agent, and this is the default. In other words, in order to operate protodep without options as before, it is necessary to set with ssh-add.

As the follows:

$ ssh-add ~/.ssh/id_rsa
$ protodep up

Getting via HTTPS

If you want to get it via HTTPS, do as follows.

$ protodep up --use-https

And also, if Basic authentication is required, do as follows. If you have 2FA enabled, specify the Personal Access Token as the password.

$ protodep up --use-https \
    --basic-auth-username=your-github-username \
    --basic-auth-password=your-github-password

License

Apache License 2.0, see LICENSE.

protodep's Issues

Support main branch

Feature: Whitelist option

If you only need a few protos from a repository that has lots of proto files in it, specifying an ignores array to exclude everything that is NOT needed is cumbersome. It would be more convenient to have a "whitelist" of files/paths that should be downloaded and ignore everything else.

Not sure if "whitelist" would be the best name for this option. Maybe "match" or "include" would be a better fit. Other ideas?

Move this repository to protodep organization

Destination: https://github.com/protodep

How to fetch repository with https basic auth

Hi

I want to fetch repository with https and basic auth.
How to do this?

Add e2e tests

Support .gitconfig

Error: clone repository: ssh: handshake failed: knownhosts: key mismatch

OS: MacOS Monterey 12.5.1
SSH Configuration seems to be fine, ssh -T [email protected] works

Remove old ignore logic

#47

Migrate to CircleCI 2.0

About the future of protodep

This project started with a little motivation from me. 4 years have passed and protodep may be in demand more than I think.
It's time to think about the future of this project.

If it's worth continuing to develop this, I need to revamp the project organization and architecture. My workload is limited and it would be better if many contributors could participate in this project.

I will consider various possibilities. Please let us know your opinion!

protodep release version doesn't have "v" prefix

• Go module requires Git version tag has "v" prefix
  ref golang/go#24007 (comment)

Currently, go install retrieve correct binary, but it shows weird version string "v0.0.0-20210821063206-6f899c69903c"

$ go install -v github.com/stormcat24/[email protected]
go: downloading github.com/stormcat24/protodep v0.0.0-20210821063206-6f899c69903c

Cloning via HTTPS fails with 'ssh: handshake failed'

I'm not sure whether I'm interpreting the code right, but it looks like protodep would support cloning of public repositories via HTTPS when it does not find a private ssh key in the users home directory. So I renamed id_rsa to id_rsa.bak and tried it with the following protodep.toml:

proto_outdir = "./protodeps"

[[dependencies]]
  target = "github.com/googleapis/api-common-protos"
  branch = "master"
  revision = "fd62e4d97ca6829b9166ae86bc6429574ff4e5db"
  path = "google/api"

Running protodep up --force then fails with this output:

[INFO] force update = true
[INFO] cleanup cache = false
[INFO] identity file = id_rsa
[INFO] use HTTP/HTTPS protocol
[INFO] Getting github.com/googleapis/api-common-protos ��Error: fetch repository is failed: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey] no supported methods remain

As I side note, I think it would make sense to allow to explicitely specify the desired protocol in protodep.toml. E.g. like this:

[[dependencies]]
  target = "https://github.com/googleapis/api-common-protos"

[[dependencies]]
  target = "ssh://github.com/googleapis/api-common-protos"

User error?

Using this config:

proto_outdir = "./proto"

[[dependencies]]
  target = "github.com/envoyproxy/envoy/"
  branch = "master"
  protocol = "https"

and creating a proto directory in the current working directory, protodep up results in:

$ protodep up
[INFO] force update = false
[INFO] cleanup cache = false
[INFO] identity file =
[INFO] use https = false
[INFO] Getting github.com/envoyproxy/envoy
[INFO] e0d7d15696c45a237e0a422f4a6f960f51bde40a is not a tag, checking out by hash

$ ls -l proto
ls: cannot access 'proto': No such file or directory

I see that it created a cache in my $homedir/.protodep, but it deletes the proto directory in my current path instead of copying the protos in the repo to it.

Any ideas on what I'm doing wrong?

Support ssh_config

dependency to the ssh identity file

Can not get from the https scheme repositories.

Trying to get the examples from the grpc-gateway:
https://github.com/grpc-ecosystem/grpc-gateway/tree/v1.4.1/examples/proto/examplepb

$ cat > protodep.toml <<EOF
proto_outdir = "./proto"

[[dependencies]]
  target = "github.com/grpc-ecosystem/grpc-gateway/examples/proto/examplepb"
  revision = "v1.4.1"
  path = "grpc-gateway/examplepb"
EOF

$ protodep up
[INFO] force update = false
[INFO] identity file = id_rsa
[INFO] use SSH protocol
[INFO] Getting github.com/grpc-ecosystem/grpc-gateway panic: x509: decryption password incorrect

goroutine 1 [running]:
github.com/stormcat24/protodep/helper.(*AuthProviderWithSSH).AuthMethod(0xc4201838e0, 0xb, 0xc420186eb0)
        $GOPATH/src/github.com/stormcat24/protodep/helper/auth.go:48 +0xa8
github.com/stormcat24/protodep/repository.(*GitHubRepository).Open(0xc4200e4780, 0xc4200e4780, 0x0, 0x0)
        $GOPATH/src/github.com/stormcat24/protodep/repository/git.go:78 +0x232
github.com/stormcat24/protodep/service.(*SyncImpl).Resolve(0xc4200b7a80, 0xc4200b7a00, 0x0, 0x0)
        $GOPATH/src/github.com/stormcat24/protodep/service/sync.go:59 +0x711
github.com/stormcat24/protodep/cmd.glob..func1(0x18e5080, 0x1908208, 0x0, 0x0, 0x0, 0x0)
        $GOPATH/src/github.com/stormcat24/protodep/cmd/up.go:61 +0x41a
github.com/spf13/cobra.(*Command).execute(0x18e5080, 0x1908208, 0x0, 0x0, 0x18e5080, 0x1908208)
        $GOPATH/src/github.com/spf13/cobra/command.go:762 +0x468
github.com/spf13/cobra.(*Command).ExecuteC(0x18e4e20, 0x1, 0xc42010ff78, 0x100620c)
        $GOPATH/src/github.com/spf13/cobra/command.go:852 +0x30a
github.com/spf13/cobra.(*Command).Execute(0x18e4e20, 0x1, 0x0)
        $GOPATH/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/stormcat24/protodep/cmd.Execute()
        $GOPATH/src/github.com/stormcat24/protodep/cmd/root.go:37 +0x2d
main.main()
        $GOPATH/src/github.com/stormcat24/protodep/main.go:6 +0x20

I have .ssh directory in $HOME.

$ test -d ~/.ssh; echo $?
0

Trying to set current directory to $HOME:

$ HOME=$(pwd) protodep up
[INFO] force update = false
[INFO] identity file = id_rsa
[INFO] use SSH protocol
[INFO] Getting github.com/grpc-ecosystem/grpc-gateway panic: open $(pwd)/.ssh/id_rsa: no such file or directory

goroutine 1 [running]:
github.com/stormcat24/protodep/helper.(*AuthProviderWithSSH).AuthMethod(0xc420183880, 0xb, 0xc420186ea0)
        $GOPATH/src/github.com/stormcat24/protodep/helper/auth.go:48 +0xa8
github.com/stormcat24/protodep/repository.(*GitHubRepository).Open(0xc4200e4780, 0xc4200e4780, 0x0, 0x0)
        $GOPATH/src/github.com/stormcat24/protodep/repository/git.go:78 +0x232
github.com/stormcat24/protodep/service.(*SyncImpl).Resolve(0xc4200b7a80, 0xc4200b7a00, 0x0, 0x0)
        $GOPATH/src/github.com/stormcat24/protodep/service/sync.go:59 +0x711
github.com/stormcat24/protodep/cmd.glob..func1(0x18e5080, 0x1908208, 0x0, 0x0, 0x0, 0x0)
        $GOPATH/src/github.com/stormcat24/protodep/cmd/up.go:61 +0x41a
github.com/spf13/cobra.(*Command).execute(0x18e5080, 0x1908208, 0x0, 0x0, 0x18e5080, 0x1908208)
        $GOPATH/src/github.com/spf13/cobra/command.go:762 +0x468
github.com/spf13/cobra.(*Command).ExecuteC(0x18e4e20, 0x1, 0xc42010df78, 0x100620c)
        $GOPATH/src/github.com/spf13/cobra/command.go:852 +0x30a
github.com/spf13/cobra.(*Command).Execute(0x18e4e20, 0x1, 0x0)
        $GOPATH/src/github.com/spf13/cobra/command.go:800 +0x2b
github.com/stormcat24/protodep/cmd.Execute()
        $GOPATH/src/github.com/stormcat24/protodep/cmd/root.go:37 +0x2d
main.main()
        $GOPATH/src/github.com/stormcat24/protodep/main.go:6 +0x20

Trying via an empty $HOME:

$ HOME= protodep up
[INFO] force update = false
[INFO] identity file = id_rsa
Error: exec: "getent": executable file not found in $PATH
Usage:
  protodep up [flags]

Flags:
  -f, --force                  update locked file and .proto vendors
  -h, --help                   help for up
  -i, --identity-file string   set the identity file for SSH (default "id_rsa")
  -p, --password string        set the password for SSH

exec: "getent": executable file not found in $PATH

Add support for ssh-agent

It looks like protodep currently requires direct access to the private ssh key. Would it be possible to add support for ssh-agent for authentication? go-git supports this cross-platform via function NewSSHAgentAuth.

Example usage: auth_method_test.go, clone.go

[Feature Request] Specify dependency by tag

I want to specify dependency by tag, but protodep seems not to specify dependency with tag.

ex: protodep.toml

proto_outdir = "./proto"

[[dependencies]]
  target = "github.com/stormcat24/protodep/protobuf"
  revision = "0.0.3"

result:

> protodep up -f

[INFO] force update = true
[INFO] identity file = id_rsa
[INFO] use SSH protocol
[INFO] Getting github.com/stormcat24/protodep
Error: get commit is failed: object not found
Usage:
  protodep up [flags]

Flags:
  -f, --force                  update locked file and .proto vendors
  -h, --help                   help for up
  -i, --identity-file string   set the identity file for SSH (default "id_rsa")
  -p, --password string        set the password for SSH

get commit is failed: object not found

Is there any way to specify dependencies by tag?

Feature: add option to suppress the spinner

We use protodep in CI builds and the spinning clock produces a lot of noise in the build logs.
Could you please add a silent mode skipping this output?

BTW: Thank you for protodep! It's really helpful

Exclude mock files in Git

Support wildcard ignores

Hey
Thanks for the great tool, its super handy.
It will be great if it would be possible to specify glob wildcards (eg *.go or */**/*.go) for the ignores directive
Currently you have to either provide a full path or the prefix which also has to be full.

Resolve nested dependencies

Hi @stormcat24, I just realised that dependencies of dependencies don't get resolved. How about checking for the existence of protodep.toml after cloning a repo and pulling in any dependencies within the same path?

Migrate to go modules

proto-dep does not support repos with any group depth

While github.com only has repos (for now) with the following structure: github.com/{owner}/{repo}.git, other hosting services (gitlab.com) can have repos with the following structure: gitlab.com/a/b/c/d/e/f.git (subgroups).

When trying to use a repo like: gitlab.com/a/b/c/d as a target, proto-dep will try connect to ssh://gitlab.com/a/b.git.

The quick fix is to change pkg/config/schema.go/Repository() to return the target as specified. Testing locally, this just works. However, this is technically a breaking change.

Protodep doesn't work with dependencies coming from multiple VCS servers

Hi, it seems that there's no convenient option to use protodep when managing dependencies coming for example from the public github and our private repositories in one protodep.toml manifest. As it uses one identity file, if provided, everywhere

Preserve path in git repository

Hi @stormcat24, first of all thanks for writing such a handy tool!

I have a use case where it would be great to preserve to directory structure within the cloned repo in my proto_outdir (e.g. github.com/googleapis/googleapis/google/type is written to ${proto_outdir}/google/type).

Basically I'm importing some dependencies which rely on github.com/googleapis/googleapis/google/type and use the import path google/type/${name}.proto, but I don't require any other files from that repo so I'm trying to avoid setting the dependency target to github.com/googleapis/googleapis and having a load of unnecessary files in my repository.

I would like to suggest adding a global option called preserve_path or similar to facilitate this.
If that's something you feel is not in the scope of protodep, an alternative solution would be allowing to specify a path on a dependency i.e.:

[[dependencies]]
  target = "github.com/googleapis/googleapis/google/type"
  branch = "master"
  path = "google/type"

I personally prefer the first method of using a global option, however the second method of specifying a path is definitely more dynamic and could help with other use-cases such as name collisions. What do you think?

Add documentation to purpose of protodep.lock

After using the tool and reading the documentation, I am unsure if protodep.lock should be checked in and the overall purpose that it serves. It is a generated file that seems to expand upon and further specify what is needed to define a dependency from the protodep.toml file. Also it seems to prevent changes to previously locked dependencies when updated in the toml file, when protodep up is called without -f. Using lock files in this was for dep management may be a common practice that is obvious to those who have run across this use of lock files, but for those like myself who aren't familiar, the documentation here is quite lacking on an explanation.
Personally I would want to always build based on the latest protodep.toml by always calling protodep up -f and disregard the protodep.lock file in source control.
I think it would help to add clarity in the README to (1) the purpose of the lock file, (2) if it should be checked in to the repo, and (3) when to force and not force update the proto deps.

Option to shell out to git?

Maybe even by default?

I went looking to do something like #49 did because protodep's ssh logic was not working well for me and certainly would not work well in a shared repo. I use per-service/host ssh keys and I would have had to specify the key to use in my build scripts. My situation isn't that unique though, some people may have transitioned to ed25519 keys or something else where the default logic is invalid.

This isn't a problem with git&ssh since I can configure git to use http/ssh per repo and git will shell out to ssh which will use the correct key and config for the host from the .ssh/config settings.

By shelling out to git #43, #41, #40 would all go away.

If this sounds good to you I'll hack on this in my spare time and submit a PR.

"worktree contains unstaged changes" error

I use following protodep.yaml.

proto_outdir = "./pbdemo"

[[dependencies]]
  target = "github.com/envoyproxy/protoc-gen-validate/validate"
  revision = "v1.0.4"
  path = "validate"
  
[[dependencies]]
  target = "github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis/google/api"
  revision = "v1.14.1"
  path = "google/api"
  
[[dependencies]]
  target = "github.com/protocolbuffers/protobuf/src/google/protobuf"
  revision = "v25.2"
  includes = [
    "descriptor.proto",
    "duration.proto",
    "empty.proto",
    "struct.proto",
    "timestamp.proto",
  ]
  path = "google/protobuf"
  
[[dependencies]]
  target = "github.com/grpc-ecosystem/grpc-gateway/protoc-gen-openapiv2/options"
  revision = "v2.19.0"
  path = "protoc-gen-openapiv2/options"

This command works fine: protodep up --use-https -f -c. But if I execute protodep up --use-https without cache clean, error checkout to 83c7c1948180f4a79579be0a13eb46e820a3ddb5: worktree contains unstaged changes apears.

Specify transport protocol in protodep.toml

#51

• git+ssh
• https://
• If neither is set.