twitchtv / retool Goto Github PK
View Code? Open in Web Editor NEWVendoring for executables written in Go
License: Apache License 2.0
Vendoring for executables written in Go
License: Apache License 2.0
eg.
https://github.com/volatiletech/sqlboiler
This command tool needs a set of template files to generate codes.
I know, things get much complicated if runtime dependencies are considered, so I am wondering is it possible to provide a way to indicate which kind of files/dir should not be dropped?
Currently, each tool is installed using go install
command which is pretty limiting. Certain tools require arguments to be passed in when building them. In order to accommodate that, the go build
command needs to be used.
Allow for passing extra parameters when building each tool which can be defined in the tools.json.
For example:
{
"Repository": "github.com/golang-migrate/migrate/cli",
"Commit": "4937cd088f7c7193ee59b319c1c2caa7482aae8e",
"ExecutableName": "migrate",
"PreParams": [
"-tags",
"'postgres'"
]
}
PreParams are the arguments passed in just after go build
command and before the path and other arguments, which in this example will result in:
go build -tags 'postgres' -o <toolDirPath>/bin/migrate github.com/golang-migrate/migrate/cli
When using retool build
with GOBIN
env variable set the produced binaries will not land where retool do
expects them to be. Instead, they will be installed to GOBIN
directory which takes precedence over $GOPATH/bin
.
To reproduce run GOBIN=/tmp/not-here retool build
. The _tools/bin
directory will not be populated like it is expected.
Proposed solution is to override both GOPATH
and GOBIN
in the install function. I'm happy to provide a PR but at a later date.
I can't easily tell what version of retool is installed. WOuld love a --version flag to show what version of retool I'm running.
Currently, each tool is installed using go install
command which will name the executable after the folder it is located in. For example, if main.go
is in .../cli/main.go
and is installed, the name of the binary will be cli
. Therefore, there is a possibility for a clash if more than one tool is contained in a folder of the same name.
Allow for (optional) custom naming of each tool set in the tools.json
If I have 2 go exes in the same vendored package, I have to refer to the folders containing the mains to get retool to vendor them. However when I add one tool then add the second (2 command line calls to retool) the second one will fail.
go version go1.16beta1 darwin/arm64
mkdir t
cd t
retool add github.com/golang/protobuf/protoc-gen-go master
retool: downloading github.com/golang/protobuf/protoc-gen-go
retool: setting version for github.com/golang/protobuf/protoc-gen-go
retool: fatal err: execution error on "git checkout ": chdir /Users/az/t/_tools/src/github.com/golang/protobuf/protoc-gen->go: no such file or directory
retool add github.com/golang/protobuf/protoc-gen-go master
retool: downloading github.com/golang/protobuf/protoc-gen-go
retool: setting version for github.com/golang/protobuf/protoc-gen-go
retool: fatal err: execution error on "git checkout ": chdir mydirectorystructure/_tools/src/github.com/golang/protobuf/protoc-gen-go: no such file or directory
but ignoring or disabling go modules temporarily seems to fix it
GO111MODULE=off retool add github.com/golang/protobuf/protoc-gen-go master
retool: downloading github.com/golang/protobuf/protoc-gen-go
retool: setting version for github.com/golang/protobuf/protoc-gen-go
retool: parsing revision "master"
retool: parsed as "d23c5127dc24889085f8ccea5c9d560a57a879d8"
retool: installing github.com/golang/protobuf/protoc-gen-go
also, creating a random directory and running retool works fine, only happens when a go.mod/sum file is present
Steps to reproduce:
mkdir test_retool
cd test_retool
retool add github.com/golang/protobuf/protoc-gen-go master
= error
In order to integrate better with dep, retool could look for packages specified in the required
section and import those into retool's configuration.
even when a tool is correctly up to date in _tools and you sync, retool goes and redownloads that tool. This is bad for several reasons:
It seems that retool currently does not analyze the dependency on .h files, thus making necessary files missing. An example of compiling with .h files is https://github.com/mattn/go-sqlite3
And maybe there are other cases the cleanup function should take care of.
Say I have a tool shovel
and a tool rake
. Both of these tools depend on different versions of package metal_head
. In the current implementation this would fail to build from a clean install simultaneously.
This is due to retool being different from the traditional vendor problem: rather than building from a single package tree, retool
can build multiple binaries from a collection of trees. In its current model these trees can share sub trees or leaves which leads to the problem described above.
I propose changing how retool collects and manipulates the GOPATH
to be a one-to-one relationship between the tool and directory. To maintain ease of use there should continue to be a single GOBIN
directory.
As a side benefit this proposal would also allow retool to not have to download all tools to sync with the manifest, as it could ask git for the current checked out sha.
Change tools dir structure, and cache dir, to be
_tools
├── bin
│ ├── shovel
│ └── rake
├── manifest.json
├── shovel
│ ├── pkg
│ │ └── linux_amd64
│ └── src
│ └── github.com
└── rake
├── pkg
│ └── linux_amd64
└── src
└── github.com
Change the install command to set GOBIN
appropriately, and keep track of each tools dir inside the tool struct.
As part of bootstrapping use of retool in our process, it would be great if there were binary releases of retool for popular platforms that we can just fetch with a curl against a versioned URL rather than needing to go get
.
I see this project is not maintained anymore, is there an alternative tool that does the same job? Maybe even one that goes a step further and vendors tools written in any language, not just Go.
It'd be useful to hook into the script generated by godownloader for repositories that publish their dependencies via https://github.com/goreleaser/goreleaser and https://github.com/goreleaser/godownloader.
What do you reckon?
When I follow the instructions in the README, I can't seem to get retool to add anything successfully. I wonder if retool doesn't work with Go modules and/or Go 1.11.2+?
In the below, notice that I'm running all this outside of $GOPATH
.
MacBook-Pro:myapp Jonathan$ go get github.com/twitchtv/retool
go: finding github.com/twitchtv/retool v1.3.7
go: downloading github.com/twitchtv/retool v1.3.7
go: finding github.com/Masterminds/semver v1.4.2
go: downloading github.com/Masterminds/semver v1.4.2
MacBook-Pro:myapp Jonathan$ retool add github.com/jteeuwen/go-bindata/go-bindata origin/master
retool: downloading github.com/jteeuwen/go-bindata/go-bindata
retool: setting version for github.com/jteeuwen/go-bindata/go-bindata
retool: fatal err: execution error on "git checkout ": chdir /private/tmp/tmpcode/myapp/_tools/src/github.com/jteeuwen/go-bindata/go-bindata: no such file or directory
MacBook-Pro:myapp Jonathan$ retool version
retool 1.3.7
MacBook-Pro:myapp Jonathan$ go version
go version go1.11.2 darwin/amd64
MacBook-Pro:myapp Jonathan$ pwd
/tmp/tmpcode/myapp
MacBook-Pro:myapp Jonathan$ echo $GOPATH
/Users/Jonathan/go
As you can see the directory it's looking for doesn't actually exist:
MacBook-Pro:myapp Jonathan$ ls -la /private/tmp/tmpcode/myapp/_tools/
total 8
drwxr-xr-x 4 Jonathan wheel 128 5 Dec 10:18 .
drwxr-xr-x 18 Jonathan wheel 576 5 Dec 10:18 ..
-rw-r--r-- 1 Jonathan wheel 27 5 Dec 10:20 .gitignore
drwxr-xr-x 3 Jonathan wheel 96 5 Dec 10:18 pkg
MacBook-Pro:myapp Jonathan$ ls -la /tmp/tmpcode/myapp/_tools/
total 8
drwxr-xr-x 4 Jonathan wheel 128 5 Dec 10:18 .
drwxr-xr-x 18 Jonathan wheel 576 5 Dec 10:18 ..
-rw-r--r-- 1 Jonathan wheel 27 5 Dec 10:20 .gitignore
drwxr-xr-x 3 Jonathan wheel 96 5 Dec 10:18 pkg
Does this seem like a bug or is it my issue?
because _tools
potentially contains Go code that isn't stored in vendor
, tooling, especially those using triple dot syntax (...
) detects vendored tools as sub-packages.
When I perform audits on code and use tree
, even blacklisting vendor
is not enough because retool replicates the source tree and my command explodes
We're starting to use this at work, and it looks awesome. However, the tools.json file only tracks commit hashes, which are not human friendly. There's no easy way to look and see what version of a tool we're requiring.
My suggestion would be to just add another field to the json, describing the requested tag/branch.
It would also be useful to have a field with the timestamp of the given commit hash, so it's more obvious if it's way out of date.
These two fields would only really be informational, the tool wouldn't need to read them at all, so I think it would be pretty easy to add (and thus would also likely be 100% backwards compatible).
I'd be happy to work on a PR if this is something that seems like it would be ok.
retool add github.com/golang/dep origin/master
retool do dep
retool: fatal err: failed on 'dep': exec: "dep": executable file not found in $PATH
Why is this happening?
I could install a dep
release to get it on my $PATH, however isn't that what retool
is trying to avoid?
Said another way, shouldn't retool add github.com/golang/dep origin/master
install the binary local to my project such that retool do dep
can find and execute it?
Using the default GOPATH
(not explicitly setting it) results in an error saying that git
is not in PATH
(it is):
> retool do
retool: syncing
retool: downloading github.com/vektra/mockery/cmd/mockery
retool: fatal err: execution error on "go get -d github.com/vektra/mockery/cmd/mockery": go: missing Git command. See https://golang.org/s/gogetcmd
package github.com/vektra/mockery/cmd/mockery: exec: "git": executable file not found in $PATH
: failed to 'go get' tool: exit status 1
It would be nice if retool worked with the default GOPATH or at least warned if GOPATH isn't set.
> retool version
retool v1.0%
Is it possible to have a -config-file
option for retool?
My issue is that I would like to follow the standard of having a tools folder for tools generated by the project rather than dev tools. I can configure the _tools directory name, but it doesn't seem like I can configure the config file name, which is awkward when there is this different concept of project-generated "tools".
Hi,
I tried to use your tool and it works without any problems with some tools I use (govendor and goveralls) except gometalinter.
As you can see, they use a _linters
directory containing the different linters they support. The problem is that directory is cleaned by the cleanup function called on tool sync, making gometalinter throwing gometalinter: error: could not find vendored linters in GOPATH
.
The only solutions I can see are:
--vendored-linters=false
, but I didn't like because it implies gometalinter to fetch linters without controlling the versionretool add --cleaning=false github.com/alecthomas/gometalinter origin/master
which generate something like this{
"Tools": [
{
"Repository": "github.com/alecthomas/gometalinter",
"Commit": "bae2f1293d092fd8167939d5108d1b025eaef9de",
"Clean": false
}
]
}
s.cleanup()
line because who care to have useless files in the _tools
directory ? It's not supposed to be in the VCSIs there another solutions ? Can you do something about this ?
Thank you.
Currently there's two sets of help, retool --help
, which prints the flags, and retool help
that prints help about the commands. At the base level, they need to be combined, so you can get a full picture of how to use the tool.
$ retool --help
Usage of retool:
-base-dir string
Path of project root. If not specified and the working directory is within a git repository, the root of the repository is used. If the working directory is not within a git repository, the working directory is used.
-f string
Use a fork of the repository rather than the default upstream
-tool-dir string
Path where tools are stored. The default value is the subdirectory of -base-dir named '_tools'.
-verbose
Enable more detailed output that may be helpful for troubleshooting.
Recently hit a bug where if the child process doesn't exit cleanly retool will leave it orphaned leaving you to force close the child process yourself.
This was hit with Terraform and is definitely an edge case when terraform doesn't cleanly exit (you need to send shutdown signal multiple times).
Currently retool
doesn't work on windows. This is due to sync.go
. The sync function uses exec.Command
to delete a directory by using rm
which is a linux specific command.
https://github.com/twitchtv/retool/blob/master/sync.go#L11
I have a fix+test, but am waiting on approval from my supervisor to open up the PR.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.