rogpeppe/gohack

Make temporary edits to your Go module dependencies

Gohack: mutable checkouts of Go module dependencies

The new Go module system is awesome. It ensures repeatable, deterministic builds of Go code. External module code is cached locally in a read-only directory, which is great for reproducibility. But if you're used to the global mutable namespace that is $GOPATH, there's an obvious question: what if I'm hacking on my program and I want to change one of those external modules?

You might want to put a sneaky log.Printf statement to find out how some internal data structure works, or perhaps try out a bug fix to see if it solves your latest problem. But since all those external modules are in read-only directories, it's hard to change them. And you really don't want to change them anyway, because that will break the integrity checking that the Go tool does when building.

Luckily the modules system provides a way around this: you can add a replace statement to the go.mod file which substitutes the contents of a directory holding a module for the readonly cached copy. You can of course do this manually, but gohack aims to make this process pain-free.

Install gohack with

go get github.com/rogpeppe/gohack

or use gobin:

gobin github.com/rogpeppe/gohack

For quick edits to a module (without version control information)

If the module to edit is example.com/foo/bar, run:

gohack get example.com/foo/bar

This will make a copy of the module into $HOME/gohack/example.com/foo/bar and add replace directives to the local go.mod file:

replace example.com/foo/bar => /home/rog/gohack/example.com/foo/bar

Note: This copy will not include version control system information so it is best for quick edits that aren't intended to land back into version control.

To edit the module with full version control

Run:

gohack get -vcs example.com/foo/bar

This will clone the module's repository to $HOME/gohack/example.com/foo/bar, check out the correct version of the source code there and add the replace directive into the local go.mod file.

Undoing replacements

Once you are done hacking and wish to revert to the immutable version, you can remove the replace statement with:

gohack undo example.com/foo/bar

or you can remove all gohack replace statements with:

gohack undo

Note that undoing a replace does not remove the external module's directory - that stays around so your changes are not lost. For example, you might wish to turn that bug fix into an upstream PR.

If you run gohack on a module that already has a directory, gohack will try to check out the current version without recreating the repository, but only if the directory is clean - it won't overwrite your changes until you've committed or undone them.

gohack's Issues

cannot undo if directory has been removed

Running any gohack command runs the go command to find out information on the current modules, but this fails if the modules cannot be resolved (for example because a gohacked module has been removed)

We should be able to undo the gohack even when the target directory no longer exists.

where should submodules be checked out?

If we try to replace two modules in the same repository, the current gohack scheme
of checking out to a single global root won't work well because the two modules
may clash (for example they may well be checked out at different versions).

One possibility might be to detect when there's a clash and use a different
name for the directory that holds the VCS directory. The problem with that
is that names become unpredictable.

cannot "upgrade" from get to get -vcs

Effectively reproduced by:

> [!net] skip
> [!exec:git] skip
> env GOPROXY=
> cd repo
> go get rsc.io/[email protected]
> env GOHACK=/gohack
> gohack get rsc.io/quote
rsc.io/quote => /gohack/rsc.io/quote
> gohack undo
> gohack get -vcs rsc.io/quote

I think this issue also raises the question of how you "downgrade" in the case you subsequently call bare get (because the assumption would be, in that case, that you aren't working in a VCS checkout).

provide way to print directory without doing anything else

It would be useful to be able to print the gohack directory for a module
without actually making any changes.

Perhaps:

gohack -p modulepath

would print the directory for modulepath.
With no arguments, -p could print the directories for all the modules in the current module that have gohack directories.

Add license

multiple subcommands

As the amount of functionality grows, it seems like we should consider having subcommands.

A possible set of commands:

 gohack get [-vcs] [-u] [-f] [module...]

Get gets the modules at the current version and adds replace statements to the go.mod file if they're not already replaced.
If the -u flag is provided, the source code will also be updated to the current version if it's clean.
If the -f flag is provided with -u, the source code will be updated even if it's not clean.
If the -vcs flag is provided, it also checks out VCS information for the modules. If the modules were already gohacked in non-VCS mode, gohack switches them to VCS mode, preserving any changes made (this might result in the directory moving).

With no module arguments and the -u flag, it will try to update all currently gohacked modules.

gohack diff module

Diff prints (in git style) changes that have been made to the module since it was checked out.

gohack rm [-f] module...

Rm removes the gohack directory if it is clean and then runs gohack undo. If the -f flag is provided, the directory is removed even if it's not clean.

 gohack undo [module...]

Undo removes the replace statements for the modules. If no modules are provided, it will undo all gohack replace statements. The gohack module directories are unaffected.

gohack dir [-vcs] [module...]

Dir prints the gohack module directory names for the given modules. If no modules are given, all the currently gohacked module directories are printed. If the -vcs flag is provided, the directory to be used in VCS mode is printed. Unlike the other subcommands, the modules don't need to be referenced by the current module.

supporting workspaces?

Would it be reasonable for gohack to support adding a module to a workspace (if a go.work file is present) rather than adding a replace to the go.mod file?

Don't pollute $HOME by default

I know the purpose of ~/gohack, but I'm not a big fan of tools that add stuff to my home directory by default.

Perhaps we could use $GOPATH/hack or something else instead. Don't have a clear answer to this one.

Add tests

I think that, for this kind of tool, the best approach would be shell-like tests. For example:

• create a new module in a temporary directory
• add a main.go with a foo/bar dependency
• run go build to update go.mod
• run gohack foo/bar
• check that the replace directive is there
• run gohack -u foo/bar
• check that the replace directive is gone

Ran gohack get github.com/user/somerepo and it is missing the .git directory

Hello.

Thought gohack was a great idea but my gohack'ed directory is missing .git. This makes it quite a bit less useful for hacking on things as I'd have to manually transfer the changes back to a version controlled directory.

Am I missing something in terms of usage? go 1.12.1 here, macOS.

Use existing dependency code source from the GOPATH

gohack should check if the dependency already exists in the GOPATH before cloning it into $HOME/gohack.

Let's say I have github.com/heetch/felice in my GOPATH.

Instead of having:

replace github.com/heetch/felice => $HOME/gohack/github.com/heetch/felice

I would have

replace github.com/heetch/felice => $GOPATH/src/github.com/heetch/felice

Add github topics

Probably go, golang, vgo, modules, vendor(?).
Something that makes gohack more discoverable over the github.
I would submit them, but there is no way to do so. :)

gohack of replaced modules does not work correctly

When a go.mod file already contains a replacement of one module by another (not a directory), gohack get -vcs checks out the original repository, not the current replacement. Also, gohack get (without -vcs) does check out the correct code, but puts it in the directory for the original module, not the replacement module, which is probably wrong.

For example, given this go.mod:

module example.com/foo

require gopkg.in/errgo.v2 v2.0.0

replace gopkg.in/errgo.v2 => github.com/rogpeppe/test2/arble v0.0.0-20181008213029-f6022c873160

If we run gohack get -vcs gopkg.in/errgo.v2, it checks out gopkg.in/errgo.v2 to $GOHACK/gopkg.in/errgo.v2, where we actually want it to check out github.com/rogpeppe/test2/arble to $GOHACK/github.com/rogpeppe/test2/arble and end up with a go.mod file like this:

module example.com/foo

require gopkg.in/errgo.v2 v2.0.0

replace gopkg.in/errgo.v2 => /home/rog/gohack/github.com/rogpeppe/test2/arble // was gopkg.in/errgo.v2 => github.com/rogpeppe/test2/arble v0.0.0-20181008213029-f6022c873160

bitbucket.org issues

I have the .gitconfig workaround for bitbucket so I can properly get private repositories on bitbucket.

[url "[email protected]:"]
insteadOf = https://bitbucket.org/

and 'gohack get bitbucket.org/xxxx' works, but I'm seeing some issue when using 'gohack get -vcs bitbucket.org/xxxx'

$ ~/go/bin/gohack get -vcs bitbucket.org/yyy/xxx-go
cannot update VCS dir for bitbucket.org/yyy/xxx-go: cannot get info: cannot find module root: https://api.bitbucket.org/2.0/repositories/yyy/xxx-go?fields=scm: 403 Forbidden

proposal: add command to determine if commit is ancestor of module version

I'm not entirely sure this belongs in gohack, but given the -vcs flag gohack makes answering it possible. So raising here for discussion at least.

I'm trying to answer the following question:

Given commit $commit, are we currently using a version of module $m that that has $commit as an ancestor?

The answer is in effect given by this sequence of commands:

GOHACK=$(mktemp -d) gohack get -vcs golang.org/x/tools
pushd $(go list -m -f {{.Replace.Dir}} golang.org/x/tools)
answer=$(git merge-base --is-ancestor $commit HEAD && echo "yes" || echo "no")
popd
gohack undo golang.org/x/tools

Does it make sense to provide a gohack command that answers this question?

cc @rogpeppe @mvdan

add 'gohack list' or shell completion support

Many package paths are long. To ease typing, I suggest either/both of:

• add gohack list, which will emit all package paths in the module, for easy copy/paste
• add shell completion, so that the tab key can do the work for me

Opinions?

cannot get module info

I am getting error

$ gohack get -vcs gopkg.in/danilopolani
cannot get module info: go list -m: can't compute 'all' using the vendor directory
	(Use -mod=mod or -mod=readonly to bypass.)

gohack does not allow relative directories in GOHACK env var.

We want to be able to set GOHACK to a relative dir, say ./. But this is not possible at the moment as the initial ./ is removed in the replace directive, creating a broken replacement line for go:

replacement module without version must be directory path (rooted or starting with ./ or ../)

We can work around the problem by manually prefixing the replacement with ./.

We like the relative path use-case, because it allows enhanced interop in WSL, when the path is on a shared Windows mount (/mnt/c/...), Windows go tools can interpret the same (relative) path as the WSL go tools. With absolute paths, Windows tools obviously cannot interpret the /mnt/c/... mount.

This is somewhat related to #27.

Idea: flag for "fork" remote

So the workflow I just used was the following:

• use gohack to pull down depedenency I want to add log statements too or changes
• cd int ~/gohack/github.com/ldelossa/repo
• added a remote called "fork" which points to my fork of github.com/ldelossa/repo
• hacked on code till I got what I wanted, pushed branch to my fork
• open PR of fork with original repo.

Obviously the fork will need to be setup everytime we run GoHack. Curious if you think it's a good idea to add a "remote" or "fork" cli flag to gohack which setups a remote fork to push your changes too in an automated fashion. If so I'd like to take a stab at this PR.

implement gohack diff subcommand

When we do gohack get of a module without specifying the -vcs flag, we record the checksum so we can later report that the source has been changed, but we have know way of informing the user exactly what has been changed.

We should record the version that's being checked out as well as the checksum. That way we can find out what's changed (using gohack diff). This also potentially allows us to switch from default to VCS mode while preserving changes made by the user.

One potentially issue with implementing this command is that it's potentially a slippery slope towards adding endless diff features.

gohack -u with no current replacements fails

When there are no currently gohacked modules, gohack fails:

% gohack -u 
failed to remove go.mod replacements: go mod edit: no flags specified (see 'go help mod edit').
error: [
	{/home/rog/src/gohack/main.go:302: failed to remove go.mod replacements}
	{/home/rog/src/gohack/exec.go:28: go mod edit: no flags specified (see 'go help mod edit').}
]

It should probably do nothing, or say "no current replacements".

add 'gohack promote' or 'gohack undo -get'

I find myself popping back and forth a lot between working on my current project and working on its dependencies.

The most painful part of this is currently when I've decided I am happy with a change in a dependency and want to use it immediately in my project. This requires:

1. commit+push dependency (not always to master, sometimes to PR for review while I continue to use it, so as not to be bloocked)
2. git rev-parse head of dependency to find version to use
3. go get pkgpath@version in project to update to latest version
4. comment out replace directive in project (or remove it; I comment out to make it easier to switch back into hack mode)

1 will always need to be done manually, but 2-4 could be automated. gohack currently only does 4.

How do you feel about adding 2 and 3 to gohack? Ideas: gohack promote or gohack undo -get or gohack undo -promote.

(I'm starting to lean towards building another tool that most closely matches my current workflow, in which I am actively working a lot on a dependency or two, but I'd like to first investigate whether I can achieve a happy union with gohack.)

gohack redo

When doing undo in the folder <somepath>, gohack should save what was undone to the file $HOME/gohack/<somepath> or to the file .gohack-redo. Command gohack redo should undo the undoing, so basically turn back what was removed (and delete redo file).

Use case in my mind is the following:

1. I'm writing the pre-commit hook, which is calling gohack undo
2. I'm writing post-commit hook, which is calling gohack redo
3. (optionally) I'm including .gohack-redo file in the .gitignore to not accidentally commit it
4. I'm working as usual, being more or less sure, that replace directives would not be commited

That would be particulary convenient for multi-repo applications, which has a shared codebase in a separate repos, because allows working on the cross-repo feature (/service1 /service2 and /lib1 /lib2 folders cloned under one root), while keeping CI builds intact (which only have /service1 cloned).

README: perhaps briefly explain how to work with a fork, including pushing a change to the fork?

In the README, maybe very briefly explain how to use a fork you created in GitHub, how to push a change back to that fork on GitHub, and then mention you can use that to open a PR on the original project?

For example, if you fork rsc.io/quote in the GitHub web interface to github.com/thepudds/quote, then use gohack + git to use that fork:

cd $(mktemp -d)                         # create an initially empty test module
go mod init testmod        
go get rsc.io/quote                     # not needed if already in go.mod

gohack get -vcs rsc.io/quote
cd $HOME/gohack/rsc.io/quote
git remote rename origin upstream
git remote add origin https://github.com/thepudds/quote
git remote -v
git checkout -b test-branch-on-my-fork 
touch new.file                          # make your edits
git add -A
git commit -am "test commit"
git push origin                         # push your changes

# then if desired, open a PR to the original project in the GitHub web interface

I purposefully modeled those steps above at least loosely on https://blog.sgmansfield.com/2016/06/working-with-forks-in-go/ because that is a somewhat common resource handed out to people who are new to Go.

@rogpeppe mentioned elsewhere that his personal workflow is usually slightly simpler than that:

cd $(mktemp -d)                         # create an initially empty test module
go mod init testmod        
go get rsc.io/quote                     # not needed if already in go.mod

gohack get -vcs rsc.io/quote
cd $HOME/gohack/rsc.io/quote/
git remote add rogpeppe https://github.com/rogpeppe/quote
git checkout -b test-branch-on-my-fork 
touch new.file                          # make your edits
git add -A
git commit -am "test commit"
git push origin

I'm not suggesting either of those be the exact text, but I am suggesting that at least mentioning a sample set of steps would be highly beneficial.

Also, people use git in many ways, so I'm not suggesting anything here be presented as "the one true way to do it", but some type of text like "There are multiple ways to do this, but one approach is ..." means someone new to modules and gohack can follow those steps, while someone more advance can map those steps to however they prefer to work with git (but doing that after having the benefit of at least understanding how gohack works with one git approach, even if it is not their personally preferred git approach).

One nuance is often someone will be using some specific version of a dependency to start in their day-to-day use (e.g., foo v1.2.3). gohack get -vcs foo will get that version by default, but that might not always be what you want if you are planning on sending a PR.

Finally, gohack probably could be tweaked to make some of this simpler, but might be easiest to start by adding something along these lines to the README based on gohack as it exists today.

Add CI

Particularly to check that go test ./... is happy, including vet. If we want this to be more than a one-man project, CI will be useful.

Happy to enable the repo on Travis and send a PR with the yaml file if @rogpeppe agrees. We can use go1.11beta3 for now.

proposal: get -tmp

The majority (based on a small sample set) of the time I find myself wanting to make throwaway changes to a dependency. In these situations I really don't want to pollute the shared $HOME/gohack space.

Hence I propose gohack get -tmp $module which would place the "hack" in a temporary directory.

Cut v1.0.0 release

TODO:

• ...

Add README

Just firing in an issue as a reminder more than anything:

• Discuss goal of gohack, that it's a playground for ideas around how to "hack" on dependencies etc
• Link to and add comments around golang/go#26640 and any other related issues
• ...

Ability to hack on an already replaced module

I think this makes sense; based on an actual scenario.

A module M I was working on had a dependency on D1. But I'd already forked D1 to D1'. Hence my go.mod already had a replace from D1 => D1'.

So my desire to hack on D1 translates to needing to hack on D1'. And hence gohack should be following replace directives to work out what should ultimately be hacked on. Instead I got:

"github.com/shurcooL/vfsgen" is already replaced; will not override replace statement in go.mod
all modules failed; not replacing anything
error: [
        {/home/myitcv/gostuff/src/gopkg.in/errgo.v2/fmt/errors/alias.go:15: all modules failed; not replacing anything}
]

Imply module name from $PWD

-*- mode: compilation; default-directory: "~/go/pkg/mod/github.com/stapelberg/[email protected]/" -*-
Compilation started at Fri May 29 21:53:41

goversion -m $(which gohack) && gohack get
/home/michael/go/bin/gohack go1.14
	path  github.com/rogpeppe/gohack
	mod   github.com/rogpeppe/gohack       v1.0.2
	dep   github.com/rogpeppe/go-internal  v1.0.0
	dep   golang.org/x/tools               v0.0.0-20180917221912-90fa682c2a6e
	dep   gopkg.in/errgo.v2                v2.1.0
cannot determine main module: go list -m: not using modules

Compilation exited abnormally with code 1 at Fri May 29 21:53:41

I have to use gohack get github.com/stapelberg/glog even though it’s obvious from my working directory which package I mean.

Would a PR for this be accepted?

add flag to avoid version reset

When we've previously checked out a version of a dependency in a gohack module dir and we want to keep that version instead of switching to the version specified in the go.mod file, it would be nice to have a flag to do that instead of running gohack and then switching back.

gohack undo leaves trailing newlines

This is how we can reproduce the issue:

$ git clone https://github.com/rogpeppe/go-internal.git
$ gohack get gopkg.in/errgo.v2
$ gohack undo
$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   go.mod

no changes added to commit (use "git add" and/or "git commit -a")

git diff produces:

diff --git a/go.mod b/go.mod
index 1c11744..e980eb2 100644
--- a/go.mod
+++ b/go.mod
@@ -3,3 +3,5 @@ module github.com/rogpeppe/go-internal
 go 1.11

 require gopkg.in/errgo.v2 v2.1.0
+
+

What I expected is that it there should be no diff, i.e.

$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

Looking at the Go's source code
https://github.com/golang/go/blob/26154f31ad6c801d8bad5ef58df1e9263c6beec7/src/cmd/go/internal/modcmd/edit.go#L209

maybe we could include modf.Cleanup() before modf.Format() in

gohack/gomodcmd.go

Line 91 in 03d2ff3

func writeModFile(modf *modfile.File) error {

Created PR #69 for this. 😄

accept package path in 'gohack get'

$ gohack get -vcs github.com/rogpeppe/go-internal/modfile
module "github.com/rogpeppe/go-internal/modfile" does not appear to be in use
all modules failed; not replacing anything

The module is github.com/rogpeppe/go-internal. But the intent of the command was clear: I want to look at the package github.com/rogpeppe/go-internal/modfile. gohack get should accept a package path and start hacking on the containing module.

well known module config

One of the more frequent comments from those moving from a GOPATH setup to modules is that they lose the ability to know where a given module dependency M is on disk. M is likely a module that they "own" / contribute to, and hence want to make changes and upstream them (frequently). The read-only module cache makes sense, and requiring a step to replace away from that remains sensible.

But I wonder whether gohack can do more to help here.

The directory $HOME/gohack is intentionally named so that it be considered a sort of throw-away area.

One of the early ideas we toyed with was "namespaced" hacks. But we settled on a sensible default for GOHACK and support for "namespaces" can be achieved by GOHACK=$HOME/mywork gohack $module.

That said, perhaps for this more common use case of "use the M on disk in its usual location" it makes sense to drive this with some sort of config.

# some sort of config for gohack such that well-known modules
# "hack" to a custom location. e.g. I might configure (a one-off operation) 
# github.com/myitcv/gobin to be in $HOME/work/gobin
#
$ gohack config github.com/myitcv/gobin $HOME/work/gobin

# Then everywhere I need to work on github.com/myitcv/gobin:
# 
$ gohack github.com/myitcv/gobin
github.com/myitcv/gobin => /home/gopher/work/gobin

Thoughts, please @rogpeppe @mvdan

getting permission denied error

Just installed gohack. Trying first hack I get:
open /var/tmp/rayj/gocode/src/github.com/lindenlab/extraction-personas/go.mod: permission denied

The repo I'm trying to hack on did get created in $HOME/gohack - it appears to be failing just opening the file. But:
ls -l go.mod
-rw-r--r-- 1 root root 3075 Nov 8 00:33 go.mod

So why is permission denied?

implement undo -rm

Currently the undo command purports to recognize the -rm and -f flags but these are ignored.

cannot update VCS dir for github.com/labstack/echo: error: pathspec 'v3.2.1+incompatible' did not match any file(s) known to git.

when run
gohack github.com/labstack/echo

it show this error message:
creating github.com/labstack/[email protected]+incompatible
cannot update VCS dir for github.com/labstack/echo: error: pathspec 'v3.2.1+incompatible' did not match any file(s) known to git.
all modules failed; not replacing anything

support linked file in gohack directory

i want to place my local change somewhere other than $HOME/gohack, so i create a linked file under it.
it reports error already exists; not overwriting when running gohack get.

could linked file be treated like a real directory?

gohack get -vcs failed with incompatible pathspec

$ gohack get -vcs k8s.io/client-go
fetching k8s.io/[email protected]+incompatible
cannot update VCS dir for k8s.io/client-go: error: pathspec 'v8.0.0+incompatible' did not match any file(s) known to git
all modules failed; not replacing anything

client-go was actually used as an example in the commit message with that pathspec format: https://go-review.googlesource.com/c/go/+/124384.

I'm hoping the fix is simple as just dropping "+incompatible" for the tag.

ability to place gohack files inside module directory instead of $HOME

I haven't tried gohack yet, but I am curious about this scenario which I have encountered when doing a similar thing manually:

I have two separate projects (let's call them myProjectA and myProjectB), each of which happen to use the github.com/jmoiron/sqlx package.

For project myProjectA, I discover a need to modify function f of sqlx, so I use gohack to get a mutable copy of the repo, and make my changes.

For project myProjectB, I want to use the original unmodified version of sqlx, but I want to place a log.Printf(...) statement inside function g of sqlx. If I were to use gohack to replace sqlx in this project, my understanding is that project myProjectB would begin using the same copy of sqlx as project myProjectA.

Is my reasoning correct?

If so, a way to avoid this seems to be to place gohack'd packages inside, say, myProjectA/.gohack/github.com/jmoiron/sqlx. This may also resolve #5?

I suppose the main issues with this are:

• Having a git repo within a git repo - however, the .gohack directory would most likely be included in .gitignore
• More difficult to use the same gohack'd package across multiple modules