dkulchenko/bunch

npm-like tool for managing Go dependencies

bunch

An npm-like tool for easily managing Go (golang) dependencies.

Overview

go get is very good. But managing multiple packages, locking versions down for reproducible builds, and changing GOPATH or rewriting package imports is an exercise left to the developer. bunch tackles this issue by providing a comprehensive tool for managing Go dependencies similar to npm for Node, supporting installing, uninstalling, pruning, listing outdated packages, and much more.

Installation

go get github.com/dkulchenko/bunch

Alternatively, precompiled binaries for supported operating systems are available.

You can pin bunch itself, if you'd like to lock the version of bunch down. If bunch finds a version of itself in the vendored environment (that is, .vendor/bin/bunch exists), it will use that version instead.

Bunchfile

See this repo's Bunchfile as an example.

github.com/this/repo !self

# comments are fun!
github.com/another/package
github.com/another/package2 v2 # can be a branch, tag, or commit
github.com/another/package3 a2b5va78d

github.com/another/package4 >= 1.0
github.com/another/package5 ~> 1.4.x
github.com/another/package6 > 1.0, < 1.4

Usage

Managing packages

(optional) Generate a Bunchfile based on your existing imports to get you started:

bunch generate

Install all packages listed in Bunchfile to .vendor directory:

bunch install

Install a specific package and save it to the Bunchfile:

bunch install github.com/abc/xyz
bunch install github.com/abc/xyz --save
bunch install github.com/abc/xyz@v2 --save
bunch install github.com/abc/xyz github.com/abc/xyz2

Update all packages to latest versions matching Bunchfile:

bunch update

Remove a package and save the change to the Bunchfile:

bunch uninstall github.com/abc/xyz
bunch uninstall github.com/abc/xyz --save

Prune unused packages from vendor directory (similar to npm prune):

bunch prune

List outdated packages:

bunch outdated

Lock down the commits currently in use (creates Bunchfile.lock; similar to npm shrinkwrap):

bunch lock

Rebuild (recompile) all packages:

bunch rebuild

Using the vendored environment

Run commands/builds within the vendored environment (sets $GOPATH and $PATH):

bunch go build .
bunch go fmt
bunch exec make
bunch shell

You can also add this to your .bash_profile to make the go command automagically be bunch-aware (e.g. go build will automatically have the correct $GOPATH set if a Bunchfile is present):

if which bunch > /dev/null; then eval "$(bunch shim -)"; fi

Limitations

For basic operations like installing/uninstalling/updating/pruning packages, all VCSes supported by go get are supported by bunch (git, hg, svn, and bzr).

For more advanced operations, packages using Git are fully supported. Mercurial has mostly full support (does not support version ranges in the Bunchfile). Bazaar has some support (does not support version ranges, "bunch outdated", or "bunch install" caching up-to-date packages). Subversion has rudimentary support (only install/update/uninstall/prune).

Contribute

Patches welcome :)

• Fork repository
• Create a feature or bugfix branch
• Open a new pull request

License

The MIT License (MIT)

Copyright (c) 2015 Daniil Kulchenko [email protected]

bunch's Issues

not fetching vendored dependencies

(sorry for the barrage of questions, I'm not sure what other channel to use...)

Is there a way to vendor dependencies that cannot be fetched on-demand (for example, because they are in private repos or in unreachable repos)? I tried to manually clone a repo into the .vendor/src tree but when I run bunch install it tries to refetch it. Basically I need a command that travis-ci can run that fetches all the normal fetch-on-demand dependencies but that doesn't die on vendored dependencies.(I use the term 'vendored' to mean dependencies that are copied into the repo as opposed to just referenced.)

`bunch generate` emits a trailing whitespace

Every line of the generated Bunchfile has a trailing space. Minor, but annoying.

`bunch` does not handle ellipsis

putting github.com/jteeuwen/go-bindata/... in Bunchfile results in the following error on bunch install:

$ bunch install
fetching github.com/jteeuwen/go-bindata/... ... ⠋ 2015/05/18 16:56:07 failed installing packages: chdir /Users/tamird/src/go/src/github.com/cockroachdb/cockroach-prod/.vendor/src/github.com/jteeuwen/go-bindata/...: no such file or directory [chdir /Users/tamird/src/go/src/github.com/cockroachdb/cockroach-prod/.vendor/src/github.com/jteeuwen/go-bindata/...: no such file or directory github.com/dkulchenko/bunch/packages.go:171:  github.com/dkulchenko/bunch/packages.go:587: ]

A bit of an edge case, but it is used by some packages that have their executables in a sub-package. See https://github.com/jteeuwen/go-bindata#installation.

bunch i installed package skipping doesn't work on multi-level packages

e.g. golang.org/x/crypto/ssh/terminal

(should scan upward for .git/.hg/etc)

Discard packages with a shared base during bunch generate

The same package can be listed multiple times in go list (e.g. github.com/a/b github.com/a/b/cli github.com/a/b/other). We should only put one of these in the generated Bunchfile.

Bunchfile.lock versions not being respected during rebuild

consider the following Bunchfile:

github.com/cockroachdb/cockroach-prod !self
github.com/awslabs/aws-sdk-go/service/ec2
github.com/awslabs/aws-sdk-go/service/elb

and the following Bunchfile.lock:

{
    "github.com/awslabs/aws-sdk-go/service/ec2": "deaa78b60cfd55045252c52f4f7840c6d697c491",
    "github.com/awslabs/aws-sdk-go/service/elb": "deaa78b60cfd55045252c52f4f7840c6d697c491"
}

Also consider:

$ pwd && git rev-parse head
/Users/tamird/src/go/src/github.com/cockroachdb/cockroach-prod/.vendor/src/github.com/awslabs/aws-sdk-go
29717a72a2fc649e790ce97dc2c4d96e32950844

In other words, the checked-out ref on disk does not match the ref specified in Bunchfile.lock. Now, consider the output of various bunch commands:

$ bunch --verbose install
skipping github.com/cockroachdb/cockroach-prod, up to date
skipping github.com/awslabs/aws-sdk-go/service/ec2, up to date
skipping github.com/awslabs/aws-sdk-go/service/elb, up to date
$ bunch --verbose rebuild
installing github.com/cockroachdb/cockroach-prod ...
successfully installed github.com/cockroachdb/cockroach-prod

installing github.com/awslabs/aws-sdk-go/service/ec2 ...
  - setting version of github.com/awslabs/aws-sdk-go/service/ec2 to  (resolved as master) ... done
  - building package github.com/awslabs/aws-sdk-go/service/ec2 ... done
  - installing package github.com/awslabs/aws-sdk-go/service/ec2 ... done
successfully installed github.com/awslabs/aws-sdk-go/service/ec2

installing github.com/awslabs/aws-sdk-go/service/elb ...
  - setting version of github.com/awslabs/aws-sdk-go/service/elb to  (resolved as master) ... done
  - building package github.com/awslabs/aws-sdk-go/service/elb ... done
  - installing package github.com/awslabs/aws-sdk-go/service/elb ... done
successfully installed github.com/awslabs/aws-sdk-go/service/elb

Am I doing something wrong? cc @mberhault

Issue running Bunch Install

Getting the following error running bunch install

failed installing packages: exit status 1 [exit status 1 github.com/dkulchenko/bunch/packages.go:439: github.com/dkulchenko/bunch/packages.go:578: ]

Bunchfile contents:

bitbucket.org/repo/project !self                                                                                                                                                                                          

gopkg.in/urfave/cli.v1                                                                                                                                                                                                                        
github.com/go-sql-driver/mysql                                                                                                                                                                                                                
github.com/Sirupsen/logrus                                                                                                                                                                                                                    
github.com/jmoiron/sqlx                                                                                                                                                                                                                       
github.com/AHaymond/go-jira/v2/jira                                                                                                                                                                                                           
gopkg.in/yaml.v2                                                                                                                                                                                                                              
github.com/fatih/structs

Consider adding GO15VENDOREXPERIMENT support

In 1.5 the go command will read the environment variable GO15VENDOREXPERIMENT. If enabled, it will use the local "/vendor/" folder for dependencies.

Will you consider supporting this approach?

`bunch prune` doesn't clean up correctly

In my project's Bunchfile:

google.golang.org/api/compute/v1
google.golang.org/api/googleapi
google.golang.org/api/resourceviews/v1beta2

Due to bunch prune bugs described in #13, all these packages are deemed unused, and are removed, along with the repo's .git. However, .vendor/src/google.golang.org/api/googleapi/ is left alone for some reason, but since the .git is gone, the next attempt to bunch install fails on go get -d google.golang.org/api/compute/v1 with:

google.golang.org/api/compute/v1: /Users/tamird/src/go/src/github.com/cockroachdb/cockroach-prod/.vendor/src/google.golang.org/api exists but /Users/tamird/src/go/src/github.com/cockroachdb/cockroach-prod/.vendor/src/google.golang.org/api/.git does not - stale checkout?

Bunch ignores dependencies of packages in cmd/...

bunch generate checks only go list --json . but some projects may contain multiple "main" packages in subdirectories. It should probably be changed to go list --json ./...

non-repo packages are not versioned properly (unknown repo type)

We're specifying packages that are not repos, and this seems to be causing problems for bunch.

Sample Bunchfile.lock line:

"github.com/awslabs/aws-sdk-go/service/ec2": "deaa78b60cfd55045252c52f4f7840c6d697c491",

bunch install output: (the interesting bit is at the end of the "setting version" line)

installing github.com/awslabs/aws-sdk-go/service/ec2 ... 
  - setting version of github.com/awslabs/aws-sdk-go/service/ec2 to  (resolved as deaa78b60cfd55045252c52f4f7840c6d697c491) ... skipped, unknown repo type
  - building package github.com/awslabs/aws-sdk-go/service/ec2 ... done
  - installing package github.com/awslabs/aws-sdk-go/service/ec2 ... done
successfully installed github.com/awslabs/aws-sdk-go/service/ec2

We're doing this a lot (full file), especially for things like the AWS SDK that have many parts we don't care about.

Please let me know if you need further information.

go get github.com/dkulchenko/bunch fails

$ go get github.com/dkulchenko/bunch
# github.com/dkulchenko/bunch
../../dkulchenko/bunch/bunch.go:53: app.Authors undefined (type *cli.App has no field or method Authors)
../../dkulchenko/bunch/bunch.go:53: undefined: cli.Author
../../dkulchenko/bunch/bunch.go:71: unknown cli.Command field 'Aliases' in struct literal
../../dkulchenko/bunch/bunch.go:89: unknown cli.Command field 'Aliases' in struct literal
../../dkulchenko/bunch/bunch.go:97: unknown cli.Command field 'Aliases' in struct literal
../../dkulchenko/bunch/bunch.go:151: unknown cli.Command field 'SkipFlagParsing' in struct literal
../../dkulchenko/bunch/bunch.go:159: unknown cli.Command field 'SkipFlagParsing' in struct literal
../../dkulchenko/bunch/bunch.go:167: unknown cli.Command field 'SkipFlagParsing' in struct literal

`bunch prune` is too aggressive

bunch prune seems not to handle sub-packages.

For instance, my project depends on github.com/cockroachdb/cockroach/util, but bunch prune only sees the top-level github.com/cockroachdb/cockroach, which is not used directly. bunch prune then removes github.com/cockroachdb/cockroach, which means I don't have github.com/cockroachdb/cockroach/util anymore, and that's Bad.

Is it possible to use `bunch` to track itself as a dependency?

I'd like to use a pinned version of bunch itself, but not sure if this is currently possible, or even theoretically possible given the bootstrapping problem. cc @bdarnell

Can bunch work with Google App Engine?

I've been using Bunch and it works great also with private repositories. Wondering if anyone has been able to get this to also work with Google App Engine?

support for private github repos

When working with a private github repo bunch fails because it tries to use the https repo URL instead of the git one. The result is that it asks for a username/password instead of using the user's SSH key. Or am I missing some trick?

Bazaar support

Code that will need to be updated:

• packages.go:getPackageRootDir
• packages.go:fetchPackage (updating existing packages)
• packages.go:setPackageVersion
• packages.go:checkPackageRecency
• versions.go:getLatestVersionMatchingPattern

Non-dependencies always get fetched

We pull in a few command-line tools for commit hooks, so they are not imported by anything, just listed in our Bunchfile.
eg:

github.com/kisielk/errcheck
github.com/barakmich/go-nyet
github.com/golang/lint/golint
golang.org/x/tools/cmd/goimports
github.com/dkulchenko/bunch

These always seem to get fetched and installed, even when up-to-date:

bunch --verbose install
refreshing github.com/kisielk/errcheck ... done
  - fetching dependencies for github.com/kisielk/errcheck ... done

refreshing github.com/barakmich/go-nyet ... done
  - fetching dependencies for github.com/barakmich/go-nyet ... done

refreshing github.com/golang/lint/golint ... done
  - fetching dependencies for github.com/golang/lint/golint ... done

refreshing golang.org/x/tools/cmd/goimports ... done
  - fetching dependencies for golang.org/x/tools/cmd/goimports ... done

refreshing github.com/dkulchenko/bunch ... done
  - fetching dependencies for github.com/dkulchenko/bunch ... done

skipping github.com/cockroachdb/cockroach-prod, up to date                 
skipping code.google.com/p/goauth2/oauth, up to date                 
skipping github.com/awslabs/aws-sdk-go/aws, up to date                 
skipping github.com/awslabs/aws-sdk-go/service/ec2, up to date                 
skipping github.com/awslabs/aws-sdk-go/service/elb, up to date                 
skipping github.com/cockroachdb/clog, up to date                 
skipping github.com/cockroachdb/cockroach/util, up to date                 
skipping github.com/ghemawat/stream, up to date                 
skipping github.com/gogo/protobuf/proto, up to date                 
skipping github.com/inconshreveable/mousetrap, up to date                 
skipping github.com/spf13/cobra, up to date                 
skipping github.com/spf13/pflag, up to date                 
skipping golang.org/x/net/context, up to date                 
skipping google.golang.org/api/compute/v1, up to date                 
skipping google.golang.org/api/googleapi, up to date                 
skipping google.golang.org/api/resourceviews/v1beta2, up to date                 
skipping gopkg.in/yaml.v1, up to date                 
installing github.com/kisielk/errcheck ... 
  - setting version of github.com/kisielk/errcheck to  (resolved as a48456c583c0111c8310fc59335f6496b8eb85f1) ... done
  - building package github.com/kisielk/errcheck ... done
  - installing package github.com/kisielk/errcheck ... done
successfully installed github.com/kisielk/errcheck                 

installing github.com/barakmich/go-nyet ... 
  - setting version of github.com/barakmich/go-nyet to  (resolved as fba7607fa3f727680833b0c44f35b448dbe5c5a8) ... done
  - building package github.com/barakmich/go-nyet ... done
  - installing package github.com/barakmich/go-nyet ... done
successfully installed github.com/barakmich/go-nyet                 

installing github.com/golang/lint/golint ... 
  - setting version of github.com/golang/lint/golint to  (resolved as 14b90a5a5501db8773a53730d1f814ccb13271f6) ... skipped, unknown repo type
  - building package github.com/golang/lint/golint ... done
  - installing package github.com/golang/lint/golint ... done
successfully installed github.com/golang/lint/golint                 

installing golang.org/x/tools/cmd/goimports ... 
  - setting version of golang.org/x/tools/cmd/goimports to  (resolved as 3d1847243ea4f07666a91110f48e79e43396603d) ... skipped, unknown repo type
  - building package golang.org/x/tools/cmd/goimports ... done
  - installing package golang.org/x/tools/cmd/goimports ... done
successfully installed golang.org/x/tools/cmd/goimports                 

installing github.com/dkulchenko/bunch ... 
  - setting version of github.com/dkulchenko/bunch to  (resolved as 85e3a193fda570b257df3fef56d278535553572f) ... done
  - building package github.com/dkulchenko/bunch ... done
  - installing package github.com/dkulchenko/bunch ... done
successfully installed github.com/dkulchenko/bunch                 

bunch -v doesn't return latest version (0.6)

Even after go get -u github.com/dkulchenko/bunch, bunch -v still shows 0.1. I think the version is hard-coded.

self link breaks code coverage

The self link (e.g. .vendor/src/github/com/org/proj -> ../../../../../) breaks code coverage profiles. At least when using ginkgo (but I don't really think plain go test is different). The .coverageprofile file produced contains file references of the form _/home/travis/gopath/src/github.com/org/proj/main.go which breaks the go cover tool.
Removing the link and adding /home/travis/gopath (in the above example) to the GOPATH makes it all work nicely. The go tool really doesn't like symlinks...

Add a !self line to bunch generated output

bunch prune deletes .git repo

When I issue bunch prune, it deletes .git repository directory.

Full Mercurial support

Code that will need to be updated:

• packages.go:getPackageRootDir
• packages.go:fetchPackage (updating existing packages)
• packages.go:setPackageVersion
• packages.go:checkPackageRecency
• versions.go:getLatestVersionMatchingPattern

bunch i should check for binaries being built before skipping package based on source being present

Finish implementing/documenting !link syntax

(Would be nice to document the bunchfile syntax fully)

bunch install gosync fails

This may not be an issue with bunch itself (I don't know).

When I issue bunch install github.com/brettweavnet/gosync, it fails with this output:

installing github.com/brettweavnet/gosync ... 2015/06/01 10:59:02 failed installing packages: failed building package github.com/brettweavnet/gosync, error: go install github.com/brettweavnet/gosync: build output "gosync" already exists and is a directory
, output: %!s(MISSING): exit status 1 [exit status 1 github.com/dkulchenko/bunch/packages.go:234: failed building package github.com/brettweavnet/gosync, error: go install github.com/brettweavnet/gosync: build output "gosync" already exists and is a directory
, output: %!s(MISSING) github.com/dkulchenko/bunch/packages.go:633: ]

I deleted .vendor folder & tried
I deleted all directories named gosync (in src, pkg, bin folders) and tried
I updated bunch and tried (bunch -v = 0.6)

Still I get the same error. As I said, I don't know if it is an error with bunch itself. But I don't know how to go about troubleshooting it. Any help?

"running vendored bin/bunch was unsuccessful" is a useless error

Can you please print the underlying error?

bunch generate ignores test dependencies

Unless I'm missing something, bunch seems to ignore dependencies in _test.go files. The result is that tests fail on CI boxes where I'm using bunch install to install all the dependencies.

`bunch lock` doesn't emit refs for non-dependencies (e.g. commands)

My project uses goimports as an executable (meaning no file depends on it) and running bunch lock emits the following into Bunchfile.lock:

"golang.org/x/tools/cmd/goimports": "",

Is this behaviour expected? Why?

Local support

Instead of just !link: absolute/path, we should include a !local so it grabs a local package and symlinks it into the .vendor directory. This is useful for when you are developing packages that are being used.

SVN support

Code that will need to be updated:

• packages.go:getPackageRootDir
• packages.go:fetchPackage (updating existing packages)
• packages.go:setPackageVersion
• packages.go:checkPackageRecency
• versions.go:getLatestVersionMatchingPattern

`bunch generate` ignores packages which are not `go get`'d

My project depends on code.google.com/p/goauth2/oauth, but bunch generate omits it, which causes bunch go build to fail. Adding "code.google.com/p/goauth2/oauth" to Bunchfile manually fixes the problem, but the entry is again removed each time bunch generate is run.

EDIT: This issue is actually that the package is not currently in my GOPATH. I have a fix on the way!

Error in installing yaml.v2

bunch install gopkg.in/yaml.v2 —save

throws the below error:

failed installing packages: failed cloning repo for package github.com/gopkg.in/yaml.v2: exit status 1

I didn't face any issue installing it via goop though.