Automate release mirroring

[alexeagle]

Automate release mirroring

[Wyverald]

This sounds interesting! Is it possible for the automation to be on the ruleset's side? (as in, every time I push a new release, send a corresponding PR to BCR) I think that makes a bit more sense than BCR having a cronjob.

re the "stretch": it's not actually much of a stretch at all -- the BCR postsubmit automatically mirrors modules to mirror.bazel.build already.

[alexeagle]

Sure, individual rulesets could set this up on their own. I think that distributes the work where it could be centralized, means copies of it will diverge. Many users won't discover it. So I think it's better done here.

[alexeagle]

Hm, we don't even collect the data for this right now.

I think the closest we have to "what GH repos to check for releases greater than X" is

jq -r --slurp '.[]|[.homepage,.versions[-1]]|@tsv' modules/*/metadata.json
https://github.com/abseil/abseil-cpp	20210324.2
https://docs.aspect.dev/bazel-lib	0.3.0
https://github.com/aspect-build/rules_js/	0.3.0
https://github.com/aspect-build/rules_swc	0.1.0
https://github.com/bazelbuild/bazel-skylib	1.0.3
https://github.com/google/boringssl	0.0.0-20211025-d4f1ab9
https://github.com/c-ares/c-ares	1.16.1
https://github.com/grpc/grpc	1.41.0
https://github.com/bazelbuild/platforms	0.0.4
https://github.com/protocolbuffers/protobuf	3.19.0
https://github.com/google/re2	2021-09-01
https://github.com/bazelbuild/rules_cc	0.0.1
https://github.com/bazelbuild/rules_java	5.0.0
https://bazelbuild.github.io/rules_nodejs/	4.4.6
https://github.com/bazelbuild/rules_pkg.git	0.5.1
https://github.com/bazelbuild/rules_proto	0.4.0
https://github.com/bazelbuild/rules_python	0.4.0
https://github.com/tweag/rules_sh#readme	0.2.0
https://github.com/bazelbuild/stardoc	0.5.0
https://github.com/protocolbuffers/upb	0.0.0-20211020-160625a
https://zlib.net	1.2.11
https://github.com/luben/zstd-jni	1.5.0-4

but some of these are docsites, links to readmes. We can try to clean up the data a bit, in most cases it's not hard to recover the github project.

I suppose another option is to correlate this with the source.json files, this data looks cleaner

jq -r --slurp '.[]|.url' modules/*/*/source.json
https://github.com/abseil/abseil-cpp/archive/refs/tags/20210324.2.zip
https://github.com/aspect-build/bazel-lib/archive/v0.3.0.tar.gz
https://github.com/aspect-build/rules_js/archive/v0.3.0.tar.gz
https://github.com/aspect-build/rules_swc/archive/v0.1.0.tar.gz
https://github.com/bazelbuild/bazel-skylib/releases/download/1.0.3/bazel-skylib-1.0.3.tar.gz
https://github.com/google/boringssl/archive/d4f1ab983065e4616319f59c723c7b9870021fae.tar.gz
https://github.com/c-ares/c-ares/releases/download/cares-1_15_0/c-ares-1.15.0.tar.gz
https://github.com/c-ares/c-ares/releases/download/cares-1_16_1/c-ares-1.16.1.tar.gz
https://github.com/grpc/grpc/archive/refs/tags/v1.41.0.zip
https://github.com/bazelbuild/platforms/releases/download/0.0.4/platforms-0.0.4.tar.gz
https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.19.0.zip
https://github.com/google/re2/archive/refs/tags/2021-09-01.zip
https://github.com/bazelbuild/rules_cc/releases/download/0.0.1/rules_cc-0.0.1.tar.gz
https://github.com/bazelbuild/rules_java/releases/download/4.0.0/rules_java-4.0.0.tar.gz
https://github.com/bazelbuild/rules_java/archive/refs/tags/5.0.0.tar.gz
https://github.com/bazelbuild/rules_nodejs/releases/download/4.4.6/rules_nodejs-core-4.4.6.tar.gz
https://github.com/bazelbuild/rules_pkg/releases/download/0.5.1/rules_pkg-0.5.1.tar.gz
https://github.com/bazelbuild/rules_proto/archive/refs/tags/4.0.0.zip
https://github.com/bazelbuild/rules_python/releases/download/0.4.0/rules_python-0.4.0.tar.gz
https://github.com/tweag/rules_sh/archive/refs/tags/v0.2.0.tar.gz
https://github.com/bazelbuild/stardoc/releases/download/0.5.0/stardoc-0.5.0.tar.gz
https://github.com/protocolbuffers/upb/archive/160625a9728b4031a21ad1e1c0146ea2c3a851eb.tar.gz
https://github.com/madler/zlib/archive/v1.2.11.zip
https://github.com/luben/zstd-jni/archive/v1.5.0-4.zip

But if we added something to metadata.json that is intended for auto-mirror then obviously we can be explicit with the data fields we need.

[aiuto]

Even if we don't auto-mirror, could we have multiple source URLs so we cna list the mirrors ourselves.

As a bonus.... if we enter a canonical github path, can the script check to see that the file is mirrored to the bazelbuild mirror, and add the second url.

[alexeagle]

I forgot about this issue - @kormide is working on it , and @meteorcloudy helped us install https://github.com/apps/publish-to-bcr as a likely way to roll this out

[meteorcloudy]

Even if we don't auto-mirror, could we have multiple source URLs so we cna list the mirrors ourselves.

I don't see why manually adding a mirror url will be better? But yes, we use http_archive under the hood, so it is possible to add this support.

[aiuto]

When I push releases I mirror them myself, so the WORKSPACE stanza that I publish looks right.
There is no need for the BCR to repeat that mirror. At a minimum, it needs to gracefully do nothing if it finds that the file is already mirrored.
And.... download data is valuable to rule owners. They might want the ability to mirror to a location where they can get download stats. We should not limit it to Google does the mirroring without a public discussion about the policy.

[meteorcloudy]

The mirror isn't hard-coded in Bazel, it's part of the configuration of the BCR, see bazel_registry.json. If a user uses modules from the BCR, they should expect a mirror configured by the BCR repo to be used. We verify the archive by the same integrity value, so nothing could go wrong. There is also the UrlRewriter in Bazel that can be used to rewrite the url if really necessary.

[meteorcloudy]

This is taken care of by https://github.com/apps/publish-to-bcr