cockroachdb/dev

A utility for performing common CockroachDB development tasks

dev's Issues

dev: munge PATH to remove ccache entries when calling into Bazel

At CRL it's very common practice to install the ccache compiler wrappers at the head of your PATH, which breaks the build:

1615575966.385574992: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /tmp
1615575966.385577832: src/main/tools/linux-sandbox-pid1.cc:279: remount rw: /dev/shm
1615575966.385598082: src/main/tools/linux-sandbox-pid1.cc:363: calling fork...
1615575966.385684892: src/main/tools/linux-sandbox-pid1.cc:393: child started with PID 2
ccache: error: Failed to create temporary file for /home/radu/.ccache/tmp/gzlib.stdout: Read-only file system
1615575966.387470242: src/main/tools/linux-sandbox-pid1.cc:410: wait returned pid=2, status=0x100
1615575966.387477002: src/main/tools/linux-sandbox-pid1.cc:428: child exited normally with code 1
1615575966.405168578: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //pkg/sql/opt/xform:xform_test failed to build
Use --verbose_failures to see the command lines of failed build steps.

The workaround is to set --sandbox_writable_path=, which is silly. ccache isn't necessary, Bazel can do the caching for us.

dev should analyze the PATH and remove any entries that look like they belong to ccache before proceeding with builds/tests.

dev: dumping ground for UX/extensions/ideas

In no particular order:

• Introduce shorthand for --filter (dev test pkg/migration -f=TestNodesString)
• Support cross-compilation
• Assert that we're operating from within a cockroachdb/cockroach checkout
• Introduce a way to build all test targets (dev build cockroach-short can fail to capture test compilation breakage)
• We could run crlfmt through dev as well (dev lint crlfmt)
• Suggest dev gen bazel (or equivalent) on the common "missing dependency" error
• (low priority) Errors with cyclical imports does not show up properly, compared to the makefile output
• dev test --stress completely hogs the CPU. We could introduce a limit for how many CPUs it can take over.
• dev gen bazel doesn't generate DEPS.bzl
• Place dev built binaries in a more convenient location (<...>/cockroachdb/cockroach/{,bin})
• If only we didn't need to specify the package name, unless needing to disambiguate (dev test --filter=TestNodesString)
• We could tie https://github.com/bazelbuild/bazelisk into the dev experience, to pin a specific bazel version
• Make use of --experimental_convenience_symlinks=ignore to not pollute the workspace with bazel-<...> symlinks
• Introduce mechanism to implant generated code from the bazel sandbox back into the source tree (#58018)
• Mock out the exec.Command layer so we can test the mapping between dev ... -> [bazel ..., ln -sf ...]; worth using datadriven for?
• dev doctor to check the working environment, and make suggestions (bazel version, autoconf, make)
• Provide an escape hatch to pass arguments to bazel directly (dev build cockroach-short --bypass '--remote_cache=...')
• Pull dev outside the crdb repo, with its own major versions. The workflow could look something like:

go install github.com/cockroachdb/dev
dev --version # v1.0, installed in $GOPATH/bin
dev build <stuff>

• Surface an easier way to override dependencies. With bazel you'd use --override_repository:

bazel build //pkg/cmd/cockroach-short --sandbox_debug \
  --override_repository=org_golang_x_tools=/Users/irfansharif/Software/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/tools

• Remove logger output that's irrelevant (the file path, info tags, etc.)
• Clear screen in the same way bazel does (the progress indicator)

$ dev build cockroach-short
I210129 21:38:36.225781 1 cmd/dev/util.go:50  executing: /usr/local/bin/bazel build --color=yes @cockroach//pkg/cmd/cockroach-short
I210129 21:38:36.414557 1 cmd/dev/util.go:65  -- Loading:
I210129 21:38:36.414783 1 cmd/dev/util.go:65  -- Loading: 0 packages loaded
I210129 21:38:36.423781 1 cmd/dev/util.go:65  -- Analyzing: target //pkg/cmd/cockroach-short:cockroach-short (0 packages loaded, 0 targets configured)
I210129 21:38:36.675993 1 cmd/dev/util.go:65  -- INFO: Analyzed target //pkg/cmd/cockroach-short:cockroach-short (7 packages loaded, 357 targets configured).
I210129 21:38:36.676061 1 cmd/dev/util.go:65  -- INFO: Found 1 target...
I210129 21:38:36.733531 1 cmd/dev/util.go:65  -- [0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
I210129 21:38:38.418549 1 cmd/dev/util.go:65  -- [68 / 297] Executing genrule //pkg/sql/parser:sql-goyacc; 1s darwin-sandbox ... (2 actions running)
I210129 21:38:43.660619 1 cmd/dev/util.go:65  -- [69 / 297] Executing genrule //pkg/sql/parser:sql-goyacc; 6s darwin-sandbox ... (2 actions, 1 running)
I210129 21:38:45.545067 1 cmd/dev/util.go:65  -- [78 / 297] GoCompilePkg pkg/sql/sem/tree/tree.a [for host]; 1s darwin-sandbox ... (2 actions running)
...
I210129 21:40:02.514410 1 cmd/dev/util.go:65  -- INFO: Elapsed time: 86.247s, Critical Path: 85.20s
I210129 21:40:02.514448 1 cmd/dev/util.go:65  -- INFO: 225 processes: 1 internal, 224 darwin-sandbox.
I210129 21:40:02.514531 1 cmd/dev/util.go:65  -- INFO: Build completed successfully, 225 total actions
I210129 21:40:02.518531 1 cmd/dev/util.go:65  --