Gazelle should fix labels in rules in Bazel-native projects imported through Go-native vendoring

[bazelbuild]

Gazelle should fix labels in rules in Bazel-native projects imported through Go-native vendoring

[jayconrod]

That seems like a useful feature. I think it's a better fit for buildozer than Gazelle though, since this is language agnostic build file maintenance.

I don't know of an existing command that does this. Would be nice to have something like:

buildozer move <from> <to> [rules...]

rules would be one or more labels for rules to update, possibly with wildcards. The command would map labels within those rules that were contained by package from to package to. So in this example:

buildozer move //vendor/github.com/bazelbuild/buildtools/... // //vendor/github.com/bazelbuild/buildtools

WDYT? If this sounds good, I'll move this issue over to bazelbuild/buildtools. If no one has bandwidth, I can work on it after the holidays.

[ianthehat]

I actually think this is the wrong approach.
I have suggested in the past that a better way to deal with vendor folders would be to use local_repository to map them to proper external entries in the workspace. This keeps their full name (including the workspace part) correct and also fixes issues like this with relative names.
Now we are starting to have gazelle write the WORKSPACE, it is more viable to have it do this for all vendor paths correctly. It does require you to tell bazel not to recurse into vendor folders, which you have to do with --deleted_packages, which you would put in your bazel.rc
It also has the benefit of preventing the top level glob targets //... from picking up vendor targets, it will only build the ones your targets depend on. Of course you could still build them by starting the glob with a workspace name.

[jayconrod]

Seems like either approach would work.

The --deleted_packages / local_repository solution is certainly simpler to set up. Gazelle would need to be told not to traverse that directory with a # gazelle:exclude directive. I think the main drawback to this approach is that it's not discoverable, both for people setting it up and people looking through those directories later.

@ixdy What's your preference on this?

[ixdy]

I did not know about --deleted_packages, but that seems potentially useful to us. We've had a number of issues with go dep including tests which don't build, since the necessary dependencies aren't included, and we've written hacky workarounds to delete the tests while waiting for an upstream fix.

--deleted_packages in a bazelrc is also more consistent with go tooling, which (IIUC) skips vendor/... on top-level globs.

Regarding local_repository - wouldn't gazelle still need to traverse the vendor directories to create BUILD files for projects that are not bazel-native?

[jayconrod]

I think for each project vendored with pre-existing build files, you'd need a --deleted_packages entry, a local_repository rule in WORKSPACE, and a # gazelle:exclude directive. Gazelle would still update build files in subtrees under vendor that weren't specifically excluded. Bazel would treat those packages normally.

[ixdy]

hm, that seems like a lot of work beyond just using go dep or a similar tool.

[jayconrod]

cc @katre

Apparently, in newer versions of Bazel, the --deleted_packages option is not necessary. Bazel will automatically exclude subtrees referenced by local_repository.

There are some other advantages to this style of vendoring: proto import paths and C++ includes don't need to be fixed. There are a lot of things that use paths relative to the repository root, and this works around those issues.

I still find it unsettling that build files in vendor subdirectories use a different labeling scheme without any apparent hint outside of the top-level WORKSPACE. It would be useful in any case to have a tool that can update paths in build files after they are moved.

[ianthehat]

One interesting pattern moving forward would be to have a vendoring repository.
This could be fetched independently, and then have it's sub folders mounted using local_repository, which would then keep the main repository clean of vendoring, while giving all the benefits current gained by vendoring, and still having the same labels as if all the children were individually mounted true bazel projects. It would also allow the same vendor repository to be shared by multiple projects within an organization.

[jayconrod]

I experimented a bit with local_repository pointing to a vendored project in Gazelle. This works well for a repository that is not a dependency of anything else. However, if you depend on a project that has vendored code like this, you need to rewrite the local_repository rules in your own workspace. I'm not sure there's a safe way to do that.

The problem is that relative paths are relative to the root directory of the main workspace, so you can't specify a relative path to a directory in another workspace. This should work better when Bazel supports recursive workspaces (presumably paths will be relative to the workspace where they are declared), but I can't recommend this approach until then.