buchgr / bazel-remote Goto Github PK
View Code? Open in Web Editor NEWA remote cache for Bazel
Home Page: https://bazel.build
License: Apache License 2.0
A remote cache for Bazel
Home Page: https://bazel.build
License: Apache License 2.0
Attempting this on Raspberry Pi 3
Command: docker run -v /tmp/bazel-cache-data:/data -p 9090:80 buchgr/bazel-remote-cache
Result:
standard_init_linux.go:190: exec user process caused "exec format error"
I am using bazel-gazelle (0.18.2) and rules_go (0.19.4) to build bazel-remote. The GoSdk version is 1.13.
The workspace repository rule is
go_repository(
name = 'com_github_buchgr_bazel_remote',
sha256 = r'''43de48c8c52078b98befe80ac51e8641f32e70c553fa91d9e4b174ee30c37788''',
strip_prefix = r'''bazel-remote''',
urls = ['http://git.corp.logiocean.com:3000/buchgr/bazel-remote/archive/c98f161a777d60b3f5dfa5748c904be764892594.tar.gz'],
importpath = r'''github.com/buchgr/bazel-remote''',
)
Then I ran
bazel build @com_github_buchgr_bazel_remote//:bazel-remote
I expect this to build correctly. But it gave me the following errors:
12:13 $ bazel build @com_github_buchgr_bazel_remote//:bazel-remote
INFO: Invocation ID: d805e18b-34b2-4e79-a601-3bd51d15ca75
INFO: Analyzed target @com_github_buchgr_bazel_remote//:bazel-remote (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
INFO: Writing explanation of rebuilds to 'build/logs'
ERROR: /home/zhongming/.cache/bazel/_bazel_zhongming/5a5efdfcd4f98edbd655a37c61128c16/external/com_github_buchgr_bazel_remote/BUILD.bazel:28:1: GoLink external/com_github_buchgr_bazel_remote/linux_amd64_static_pure_stripped/bazel-remote failed (Exit 1) builder failed: error executing command
(cd /home/zhongming/.cache/bazel/_bazel_zhongming/5a5efdfcd4f98edbd655a37c61128c16/sandbox/linux-sandbox/32199/execroot/logi && \
exec env - \
CGO_ENABLED=0 \
GOARCH=amd64 \
GOOS=linux \
GOROOT=bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib% \
GOROOT_FINAL=GOROOT \
PATH=tools/toolchains/cc/clang/wrappers:/bin:/usr/bin \
bazel-out/host/bin/external/go_sdk/builder link -sdk external/go_sdk -installsuffix linux_amd64 -arc '@com_github_buchgr_bazel_remote//cache:go_default_library=github.com/buchgr/bazel-remote/cache=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache.a' -arc '@com_github_buchgr_bazel_remote//cache/disk:go_default_library=github.com/buchgr/bazel-remote/cache/disk=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/disk/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache/disk.a' -arc '@com_github_djherbis_atime//:go_default_library=github.com/djherbis/atime=bazel-out/k8-fastbuild/bin/external/com_github_djherbis_atime/linux_amd64_static_pure_stripped/go_default_library%/github.com/djherbis/atime.a' -arc '@com_github_buchgr_bazel_remote//cache/gcs:go_default_library=github.com/buchgr/bazel-remote/cache/gcs=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/gcs/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache/gcs.a' -arc '@com_github_buchgr_bazel_remote//cache/http:go_default_library=github.com/buchgr/bazel-remote/cache/http=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/cache/http/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/cache/http.a' -arc '@org_golang_x_oauth2//:go_default_library=golang.org/x/oauth2=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2.a' -arc '@org_golang_x_oauth2//internal:go_default_library=golang.org/x/oauth2/internal=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/internal/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/internal.a' -arc '@org_golang_x_net//context/ctxhttp:go_default_library=golang.org/x/net/context/ctxhttp=bazel-out/k8-fastbuild/bin/external/org_golang_x_net/context/ctxhttp/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/net/context/ctxhttp.a' -arc '@org_golang_x_oauth2//google:go_default_library=golang.org/x/oauth2/google=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/google/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/google.a' -arc '@org_golang_x_oauth2//jws:go_default_library=golang.org/x/oauth2/jws=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/jws/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/jws.a' -arc '@org_golang_x_oauth2//jwt:go_default_library=golang.org/x/oauth2/jwt=bazel-out/k8-fastbuild/bin/external/org_golang_x_oauth2/jwt/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/oauth2/jwt.a' -arc '@com_google_cloud_go//compute/metadata:go_default_library=cloud.google.com/go/compute/metadata=bazel-out/k8-fastbuild/bin/external/com_google_cloud_go/compute/metadata/linux_amd64_static_pure_stripped/go_default_library%/cloud.google.com/go/compute/metadata.a' -arc '@com_github_buchgr_bazel_remote//config:go_default_library=github.com/buchgr/bazel-remote/config=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/config/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/config.a' -arc '@in_gopkg_yaml_v2//:go_default_library=gopkg.in/yaml.v2=bazel-out/k8-fastbuild/bin/external/in_gopkg_yaml_v2/linux_amd64_static_pure_stripped/go_default_library%/gopkg.in/yaml.v2.a' -arc '@com_github_buchgr_bazel_remote//server:go_default_library=github.com/buchgr/bazel-remote/server=bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/server/linux_amd64_static_pure_stripped/go_default_library%/github.com/buchgr/bazel-remote/server.a' -arc '@com_github_abbot_go_http_auth//:go_default_library=github.com/abbot/go-http-auth=bazel-out/k8-fastbuild/bin/external/com_github_abbot_go_http_auth/linux_amd64_static_pure_stripped/go_default_library%/github.com/abbot/go-http-auth.a' -arc '@org_golang_x_crypto//bcrypt:go_default_library=golang.org/x/crypto/bcrypt=bazel-out/k8-fastbuild/bin/external/org_golang_x_crypto/bcrypt/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/crypto/bcrypt.a' -arc '@org_golang_x_crypto//blowfish:go_default_library=golang.org/x/crypto/blowfish=bazel-out/k8-fastbuild/bin/external/org_golang_x_crypto/blowfish/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/crypto/blowfish.a' -arc '@org_golang_x_net//context:go_default_library=golang.org/x/net/context=bazel-out/k8-fastbuild/bin/external/org_golang_x_net/context/linux_amd64_static_pure_stripped/go_default_library%/golang.org/x/net/context.a' -arc '@com_github_urfave_cli//:go_default_library=github.com/urfave/cli=bazel-out/k8-fastbuild/bin/external/com_github_urfave_cli/linux_amd64_static_pure_stripped/go_default_library%/github.com/urfave/cli.a' -package_list bazel-out/host/bin/external/go_sdk/packages.txt -X 'github.com/buchgr/bazel-remote/server.GitCommit={STABLE_GIT_COMMIT}' -o bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/linux_amd64_static_pure_stripped/bazel-remote -main bazel-out/k8-fastbuild/bin/external/com_github_buchgr_bazel_remote/linux_amd64_static_pure_stripped/bazel-remote%/github.com/buchgr/bazel-remote.a -p github.com/buchgr/bazel-remote -- -extld tools/toolchains/cc/clang/wrappers/clang -linkmode external '-buildid=redacted' -w -extldflags '-Lexternal/clang/lib -lc++abi -lm -fuse-ld=lld -Wl,-z,relro,-z,now -no-canonical-prefixes -static')
Execution platform: @bazel_tools//platforms:host_platform
Use --sandbox_debug to see verbose messages from the sandbox
external/go_sdk/pkg/tool/linux_amd64/link: running tools/toolchains/cc/clang/wrappers/clang failed: exit status 1
ld.lld: error: undefined symbol: x_cgo_callers
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_callers)
ld.lld: error: undefined symbol: x_cgo_init
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_init)
ld.lld: error: undefined symbol: x_cgo_mmap
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_mmap)
ld.lld: error: undefined symbol: x_cgo_munmap
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_munmap)
ld.lld: error: undefined symbol: x_cgo_notify_runtime_init_done
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_notify_runtime_init_done)
ld.lld: error: undefined symbol: x_cgo_sigaction
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_sigaction)
ld.lld: error: undefined symbol: x_cgo_thread_start
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(_cgo_thread_start)
ld.lld: error: undefined symbol: x_cgo_setenv
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(runtime._cgo_setenv)
ld.lld: error: undefined symbol: x_cgo_unsetenv
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(runtime._cgo_unsetenv)
ld.lld: error: undefined symbol: _cgo_yield
>>> referenced by go.go
>>> /tmp/go-link-481443378/go.o:(runtime.cgo_yield)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
link: error running subcommand: exit status 2
Target @com_github_buchgr_bazel_remote//:bazel-remote failed to build
INFO: Elapsed time: 0.564s, Critical Path: 0.38s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
I have been building this binary using GoSdk 1.11 through 1.12. Those versions (with their compatible versions of rules_go and bazel-gazelle) of GoSdk built bazel-remote
correctly.
I know very little about cgo. But this error looks strange to me as I did not expect bazel-remote to have any cgo code.
Help or pointers are appreciated.
Thanks!
The cache should only garbage collect action cache entries based on last access time. CAS entries should have a reference count and only be collected if they are no longer referenced by any cache entry.
Hello,
We are investigating on using Bazel as our build system, and are assessing using bazel-remote as our remote caching. However, I am concerned that the Bazel's official documentation lists this as experimental
and un-supported
.
Bazel Remote Cache is an open source remote build cache that you can use on your infrastructure. It is experimental and unsupported.
Can it be confirmed that this is not experimental
? I do see that there are enough commits in the past few months that it is being supported.
This results in errors similar to:
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET-class.jar' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET.jar_manifest_proto' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET.jdeps' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: output '$PACKAGE/lib$TARGET-native-header.jar' was not created
ERROR: $WORKSPACE/$PACKAGE/BUILD:1:1: not all outputs were created or valid
when using java rules for example.
Incompatible flag --incompatible_no_implicit_file_export will break Bazel Remote Cache once Bazel 1.2.1 is released.
Please see the following CI builds for more information:
Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration
Important: Please do NOT modify the issue title since that might break our tools.
One of the most important features for any remote cache (IMO) is metrics. We want to know how healthy the cache is and be notified if something changes.
Adding support for Prometheus would give us that functionality.
We found that on some hardware this absolutely kills the performance of the system e.g. a Mac Pro became completely unresponsive. In general, there is no need for calling Sync() on every write as in the worst case it's ok to lose data. We should only ensure that on loading the entries from disk that there is data integrity.
Probing http://mybazelremote.local/ returns error 400 causing Kubernetes liveness and readiness checks to fail. What path can we use to probe the server's health? eg. http://mybazelremote.local/healthz
Thanks.
Bazel remote caching has issues with files updating during build. To avoid them I would like to disable writing to remote cache for desktop machines and only allow the CI server to do the writing. There is a bazel flag build --remote_upload_local_results=false
that can help me, but developers can still accidentally write to the remote cache. Hence it's best to have two instances of the cache over the same data directory and make one of them read-only to be used by developers and another one read-write for the CI server to cache the "master" branch.
I propose adding a -read-only
flag that will allow starting a read-only cache instance that will discard PUT
requests.
I think the server should also:
.gz
file without decompressing.Object files compress very well, so this will save significant amount of IO/bandwidth. What do you think @buchgr?
Incompatible flag --incompatible_load_python_rules_from_bzl will break Bazel Remote Cache once Bazel 1.2.1 is released.
Please see the following CI builds for more information:
Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration
Important: Please do NOT modify the issue title since that might break our tools.
We had a case of cache poisoning where to fix, we attempted to delete the bad file, this resulted in this server error:
WARNING: Reading from Remote Cache:
500 Internal Server Error
stat /data/cas/a25641977eb4f12d165dda1a1459c1c8c449ffb1146b90cfec5176c1ca85f3db: no such file or directory
It would be nice to handle this gracefully and 404
I think it would be helpful if the README.md described the REST API, and the difference between the action cache and the CAS cache.
After poking around a bit I found some info in #5, if you like I can have a go at documenting this?
The error, described at the end, occurs when trying to build from source
machine on centos7
compiler is gcc 8.3.0 installed not local to the machine
I was needing LD_LIBRARY_PATH to pass this but this step eas not taking that into account.
I am new to bazel, I found out that option "use_default_shell_env=True" in actions.run
was necessary.
a modification of external/io_bazel_rules_go/proto/compiler.bzl in actions.run fixed the problem
bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/go-protoc -protoc bazel-out/host/bin/external/com_google_protobuf/protoc -importpath google.golang.org/genproto/protobuf/api -out_path bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/api_go_proto%/ -plugin bazel-out/host/bin/external/com_github_golang_protobuf/protoc-gen-go/linux_amd64_stripped/protoc-gen-go -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/source_context_proto-descriptor-set.proto.bin -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/any_proto-descriptor-set.proto.bin -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/type_proto-descriptor-set.proto.bin -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/api_proto-descriptor-set.proto.bin -expected bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/api_go_proto%/google.golang.org/genproto/protobuf/api/api.pb.go -import 'google/protobuf/source_context.proto=google.golang.org/genproto/protobuf/source_context' -import 'google/protobuf/any.proto=github.com/golang/protobuf/ptypes/any' -import 'google/protobuf/type.proto=google.golang.org/genproto/protobuf/ptype' -import 'google/protobuf/api.proto=google.golang.org/genproto/protobuf/api' google/protobuf/api.proto)
ERROR: /net/hogsback/home1/thierry/bazel/bazel/_bazel_thierry/aa4e8273d17ebfffbcef68d297df2765/external/io_bazel_rules_go/proto/wkt/BUILD.bazel:3:1: Generating into bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/struct_go_proto%/github.com/golang/protobuf/ptypes/struct failed (Exit 1) go-protoc failed: error executing command
(cd /net/hogsback/home1/thierry/bazel/bazel/_bazel_thierry/aa4e8273d17ebfffbcef68d297df2765/sandbox/processwrapper-sandbox/915/execroot/main &&
exec env -
CGO_ENABLED=1
GOARCH=amd64
GOOS=linux
GOROOT=external/go_sdk
GOROOT_FINAL=GOROOT
PATH=/soft/fw/tools/gcc/gcc-8.3.0-rhel7-x64/bin:/usr/bin:/bin
bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/go-protoc -protoc bazel-out/host/bin/external/com_google_protobuf/protoc -importpath github.com/golang/protobuf/ptypes/struct -out_path bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/struct_go_proto%/ -plugin bazel-out/host/bin/external/com_github_golang_protobuf/protoc-gen-go/linux_amd64_stripped/protoc-gen-go -descriptor_set bazel-out/k8-fastbuild/bin/external/com_google_protobuf/struct_proto-descriptor-set.proto.bin -expected bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/proto/wkt/linux_amd64_stripped/struct_go_proto%/github.com/golang/protobuf/ptypes/struct/struct.pb.go -import 'google/protobuf/struct.proto=github.com/golang/protobuf/ptypes/struct' google/protobuf/struct.proto)
Execution platform: @local_config_platform//:host
Use --sandbox_debug to see verbose messages from the sandbox
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc) bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version
GLIBCXX_3.4.21' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
bazel-out/host/bin/external/com_google_protobuf/protoc: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by bazel-out/host/bin/external/com_google_protobuf/protoc)
2019/12/04 19:28:24 error running protoc: exit status 1
Target //:bazel-remote failed to build
INFO: Elapsed time: 1.239s, Critical Path: 0.42s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
Such as current size, number of objects, etc.
Hi, just wondering if this functionality is intended.
If I use the image you provide (buchgr/bazel-remote-cache
), everything works fine.
If I built it myself, it populates /data/bazel-cache
with the following:
How I build the image:
# Build image
$ bazel run :bazel-remote-image -- --norun
# Run container
$ docker run -v /data/bazel-cache:/data -p 8080:8080 bazel:bazel-remote-image
I can't seem to build this project. I get the following error when running bazel build ...
:
compilepkg: missing strict dependencies: ... import of "golang.org/x/sys/unix"
No dependencies were provided.
Check that imports in Go sources match importpath attributes in deps.
When looking up Action messages stored in the ActionCache, the results are only usable if all the referenced output blobs are also available in the CAS. When the client performs a GET or HEAD request, bazel-remote should confirm that these blobs exist (thereby moving them to the "safe" end of the LRU queue), and otherwise return 404 (even if the Action exists).
Since adding extra validation might break existing users (eg if using bazel-remote's ActionCache as a generic blob store), perhaps we should add a new raw/ REST API endpoint that behaves like ac/ does currently (ie no validation). I'm not sure if there's a reasonable automatic migration path, but sysadmins can at least rename preexisting ac/ directories to raw/ before upgrading bazel-remote.
Another option would be to add a command line flag to turn on extra validation for HTTP ac/.
I feel like caching metadata in memory would be super efficient. Is it worth implementing? @buchgr
Is there any reason the cache directory defaults to 0744 instead of the more common 0755 or even respecting umask
?
I'm talking about this code.
Include a brief introduction of how to set up the development environment, what's the recommended IDE, how to run test inside IDE, etc..
I tried to set up running test inside Intellij but did not figure out. I just ended up running test in command line with bazel
This is desirable for enterprise use.
I am not convinced that we the remote cache should verify the hash sum of CAS entries, in that it costs significant processing power and it's not clear that it helps with anything :-). Wdyt @nicolov?
It seems like the AC and CAS hashes could collide since they both appear to be stored in <cache>/<hash>
.
Perhaps instead we could store them at <cache>/ac-<hash>
and <cache>/cas-<hash>
?
On startup, bazel-remote scans the cache directory and sorts cache files by their atime. The Get and Put functions update the in-memory LRU index and open the file which updates the atime, but the Contains function only updates the in-memory LRU index. When bazel-remote restarts, the in-memory LRU index is lost and the new index that is recalculated will miss the effects of Contains.
Should bazel-remote update the atimes of cache files when Contains is called? This would give more correct results after restarts, but might be slower. Or should we implement a graceful shutdown process that flushes the in-memory index to disk (either by touching all the cache files in LRU order, or by explicitly dumping the in-memory LRU to disk and using that at startup time)?
We don't specify a particular version, and I think the default changed from v1 to v2 when v2.0.0 was released. Let's fix this by upgrading to v2.
Related to #59, I have a setup where developer machines have a local cache they write to, as discussed in bazelbuild/bazel#5139, however to proxy to the shared cache that CI writes to one must also enable writing to it, as there is no configuration knob for that. I could disable writing to the cache on the client side, but then we wouldn't be using the cache running locally as a directory cache.
I think there are some bugs in EnsureSpacer, as my disks keep filling up. Anyways, I've been reading the code and I don't understand why it uses the low/high water marks to trigger deletion. We could store and update the atimes in cache.files
, and ensure that there's enough space before each upload, instead of doing it in bulk once the cache goes over the high water mark. This would ensure optimal disk usage.
In the same way that Bazel provides binaries, bazel-remote
should also provide binary releases.
One of the uses of the remote cache is as a proxy on local machines (Bazel waits for all the artifacts to upload before completing the build so a slow network connection can impact build times).
I've been playing with the idea of having my Bazel wrapper automatically spawn a bazel-remote
subprocess on a local port and rewire the Bazel command to prefer the local cache server than the remote one.
In order for this to work, there are two features that I think would be needed from bazel-remote
:
bazel-remote
process is already serving objects from a directory.I think for 1. the easiest would be to write the PID and port number in a file inside the cache directory. On startup, bazel-remote
could check if that PID is still running, if not, it would start on the same port.
I could manage this from the Bazel wrapper, but I think it would be better to add it to bazel-remote
so I'm opening an issue to start the discussion.
I've been testing out bazel-remote
and noticed that the disk usage exceeds the max_size
. It seems like the larger I set the max to, the larger the error:
max_size=5
actual disk usage of cas dir is >5.1gb
max_size=180
actual disk usage of cas dir >182gb
This was surprising for me and results in 500 errors. In testing, I set max_size=199
where the volume /data
was only~200gb
in size. bazel-remote
ended up returning 500
on PUT
since the total usage exceeded the available space, so I'm not sure what size it would have grown to.
I like the bazel-remote project, but storing the files on disk may not scale for our needs.
Some of the remote execution implementations (such as build-grid and build-barn) also allow storing the cache objects in S3.
I'd be nice for bazel-remote to have the same capability.
With bazel-remote
running on localhost and an http_proxy
configured to a remote bazel-remote
, when an object is looked up remotely and not found, the local bazel-remote
responds with a 500 Internal Server error.
This is the output from Bazel (0.22.0):
$ bazel test //...
INFO: Using Bazel Remote listening on port 55381
INFO: Running Bazel version 0.22.0.4
INFO: Invocation ID: 243c2863-f4f3-4a27-b35c-8ee3362b4990
INFO: Analysed 40 targets (0 packages loaded, 0 targets configured).
INFO: Found 38 targets and 2 test targets...
WARNING: Error reading from the remote cache:
500 Internal Server Error
Not found
INFO: Elapsed time: 0.886s, Critical Path: 0.63s
INFO: 1 process: 1 darwin-sandbox.
INFO: Build completed successfully, 2 total actions
There are no obvious errors in the local bazel-remote
log, but I used tcpdump
to see the packets:
This is Bazel:
GET /ac/5a776703f6b95900eba8e7a1779df127f0c61a60610e9e1322c94d3bbe3bd1fd HTTP/1.1
host: 127.0.0.1:55381
connection: keep-alive
accept: */*
This is the local bazel-remote
log:
2019/03/01 19:27:46 GET 404 http://bazel-remote.example.com:8415/ac/5a776703f6b95900eba8e7a1779df127f0c61a60610e9e1322c94d3bbe3bd1fd
2019/03/01 19:27:46 GET /ac/5a776703f6b95900eba8e7a1779df127f0c61a60610e9e1322c94d3bbe3bd1fd: Not found
This is the response from the local bazel-remote
:
HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Fri, 01 Mar 2019 19:27:46 GMT
Content-Length: 11
Not found
The local bazel-remote
should reply with an HTTP code 404 Not Found instead of a 500 Internal Server Error.
@BenTheElder mentioned to me that k8s is using a different remote cache for different projects and they distinguish between projects via a path component in the URL:
https://cache.com/<project name>/(ac|cas)/
I think that's a good idea and I believe @nicolov already laid the foundations for that.
In this line, it's necessary to read the response and close the body, like so:
@@ -68,6 +68,10 @@ func uploadFile(remote *http.Client, baseURL *url.URL, local cache.Cache, access
if err != nil {
return
}
+ ioutil.ReadAll(rsp.Body)
+ rsp.Body.Close()
+
+
logResponse(accessLogger, "PUT", rsp.StatusCode, url)
return
}
or bazel-remote
will leak file descriptors when uploading to a remote cache.
From the Go documentation:
// If the returned error is nil, the Response will contain a non-nil
// Body which the user is expected to close. If the Body is not both
// read to EOF and closed, the Client's underlying RoundTripper
// (typically Transport) may not be able to re-use a persistent TCP
// connection to the server for a subsequent "keep-alive" request.
//
// The request Body, if non-nil, will be closed by the underlying
// Transport, even on errors.
It would be great if we could add a way to use this cache as a library (i.e. to make integration into existing projects easier). As a side effect, this would mostly implement #12, except the configuration part, and make it easier for folks to implement their own storage mechanism on top of GCS, S3, ...
We would create a new package httpcache
(to avoid conflicts with net/http
), which has a function New
that returns a http.Handler
.
type Options struct {
// Logging, ...
}
// New creates a new Bazel cache which uses the provided StorageMechanism
// to store and retrieve cached files.
func New(StorageMechanism, Options) http.Handler {
// ...
}
(I think StorageMechanism
will look mostly like LRU
@nicolov implemented recently).
WDYT?
I was thinking of creating an Http middleware to do some logging. @buchgr did you have other ideas in mind?
This repository cannot build with Bazel incompatible flags.
At least --incompatible_no_transitive_loads
is failing, probably more.
bazelisk --migrate
can be useful to find the issues.
Could you fix the issues?
Thanks
Not sure if that is possible, but it would be nice to somehow protect external dependencies from being automatically deleted. The idea behind this is to be able to still enforce a maximum size for project targets but be guaranteed to have your external dependencies at all times and therefore have some protection against the case of third party dependencies having been removed or internet downtime (given the server being in the local network).
Let me know what you think of this idea.
I recently added some ActionResult validation, including one check that seems to be technically incorrect but still useful in practice: zero-length serialized ActionResult messages are currently considered invalid.
This is technically incorrect, because an ActionResult might legitimately have no outputs apart from the exit code (and if that exit code is 0 then the serialized value could be zero-length). One possible use case for this is a sanity check. But this is bad, because a zero-length file is perhaps the most common form of data corruption, and we have no way to recognise when this is corruption and when it is legitimate.
However in practice, there are many cases where this type of ActionResult can be reasonably be expected never to occur (eg when REAPI is only used for compilation jobs, like recc or goma jobs). And in that case, maybe it's worth keeping this validation, off by default and hidden by a command line flag. Would such a flag be welcome, or should I just remove this validation?
It's currently single-threaded and can be quite slow for large caches.
Right now bazel-remote
just logs to stdout/stderr, which means it needs to be spawn by redirecting those to a file and the log files managed externally.
Alternatively, it could use something like lumberjack and rotate the logs itself automatically. It would need some new configuration parameters to determine log location (the default would be under the cache directory), and how often to rotate the logs.
I propose the following new config options:
log_path: path.log # optional, if relative path it goes under cacheDir, defaults to bazel-remote.log
log_max_size: 500 # in megabytes
log_max_backups: 3 # number of backup files to keep around
log_max_age: 28 # how many days to keep the around for
log_compress: true # disabled by default
I was running your amazing cache with a very small cache size and noticed that new files were no longer added to the cache after some time. After reading the code, I think there are a few bugs in EnsureSpacer
:
d, err := os.Open(cache.Dir())
// ...
files, err := d.Readdir(-1)
// ...
for _, fileinfo := range files {
name := fileinfo.Name()
// ...
}
d
is never closed. I think I've seen some errors related to too many files open in log files, but I'm not 100% sure that it is related to this.files
will always be a slice containing exactly ac
and cas
. I think what we want to do here is find the oldest files in the cache and delete them. This could be done with filepath.Walk
, similar to what we do in LoadExistingFiles()
, but since we know what files are stored in the cache, we could maybe find a more efficient solution.fileinfo.Name()
returns the base name of a file, not the path (https://golang.org/pkg/os/#FileInfo). Unfortunately, there is no way to get the full path from FileInfo.I think we could add support for debugging cache misses between two machines to the remote cache.
This can detect and help fix three kinds of errors:
Thoughts?
I'm seeing errors like:
process bazel-remote-0.0[61853] caught waking the CPU 45001 times over ~42 seconds, averaging 1066 wakes / second and violating a limit of 45000 wakes over 300 seconds.
my configuration uses idle_timeout
so I suspect we are hitting a Go runtime bug: golang/go#27707
Still investigating but wanted to file the issue in case anyone hits it or has any ideas.
We have a case where some empty actions managed to sneak into our ac
cache.
$ find ac -size 0 | head
ac/0a/0a372b8f6f57e408155b2d77291599fa69c3ad44c32a370579bc3fd582f462e6
ac/1d/1d16b14f8365aa77a51617086c6a6675fd216480223ffa02a5cbf33322570f8d
ac/87/875ac9c5592f5f8c86dc05d3c61d18a9d7aac85c0930b7b36290ba9f46fa3367
These are problematic because Bazel will fail the build when the outputs are not created.
Would it make sense for bazel-remote
to refuse to serve empty actions?
I'm able to pull down the latest code, build it, and run it (admittedly without exhaustive testing), but was curious if anyone would be pushing the latest to DockerHub in the near future.
We should probably not specify username and password on the command line but instead support something like the .htpasswd
format that's used by web servers for basic authentication.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.