bazelbuild / bazel-federation Goto Github PK

View Code? Open in Web Editor NEW

47.0 47.0 18.0 235 KB

License: Apache License 2.0

Python 30.32% Go 2.77% Rust 2.58% Scala 2.77% Java 0.35% C++ 0.34% Starlark 60.88%

bazel-federation's People

Contributors

Stargazers

Watchers

Forkers

laurentlb dslomov renovate-bot nlopezgi fweikert aiuto yannic brandjon jjardon meteorcloudy tomaszstrejczek mishudark achew22 isabella232 aranguyen ungtb10d

bazel-federation's Issues

Flag --incompatible_use_platforms_repo_for_constraints will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_use_platforms_repo_for_constraints will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (no JDK)](:ubuntu: 18.04 (no JDK))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Python not working

When I try to use federation via:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "bazel_federation",
    url = "https://github.com/bazelbuild/bazel-federation/archive/130c84ec6d60f31b711400e8445a8d0d4a2b5de8.zip",
    sha256 = "9d4fdf7cc533af0b50f7dd8e58bea85df3b4454b7ae00056d7090eb98e3515cc",
    strip_prefix = "bazel-federation-130c84ec6d60f31b711400e8445a8d0d4a2b5de8",
    type = "zip",
)

load("@bazel_federation//setup:rules_python.bzl", "rules_python_setup")
rules_python_setup()

I get an error:

ERROR: Failed to load Starlark extension '@rules_python//python:pip.bzl'.
Cycle in the workspace file detected. This indicates that a repository is used prior to being defined.
The following chain of repository dependencies lead to the missing definition.
 - @rules_python
This could either mean you have to add the '@rules_python' repository with a statement like `http_archive` in your WORKSPACE file (note that transitive dependencies are not added automatically), or move an existing definition earlier in your WORKSPACE file.
ERROR: cycles detected during target parsing

repositories.bzl should check bazel_version and ensure minimum bazel

There is a federation track for each incompatible bazel release.

I believe it is a fair tradeoff to required that the federation tested and working against bazel 1.0.n is restricted to bazel 1.*. The condition I want to prevent is

user has bazel 0.29.0
bazel is at bazel 1.0.0
federation 42.0.0 is tested with bazel 1.0.0 - but it also happens to actually work with bazel 0.29
user uses federation 42.0.0, and additional rule sets that are not in federation
federation updates a component and that component now requires bazel 1.0.0. This is now federation 42.1.0.
user updates to fedeation 42.1.0 but now their code does not work

In federation semantic versioning, it is correct to go to 42.1.0, because we are not changing the advertised features in an incompatible way. But we are changing the unadvertised features, and breaking the user that way.

Implementation thoughts

We have native.bazel_version. Great.
@bazel_skylib has version compare macros. Great.
Do we make Federation take a dependency on skylib?
- Yes. But I think I want to unwind also having skylib depend on the federation. The development is too hard when they are intertwined.

Flag --incompatible_load_python_rules_from_bzl will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_load_python_rules_from_bzl will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (no JDK)](:ubuntu: 18.04 (no JDK))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Flag --incompatible_load_proto_rules_from_bzl will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_load_proto_rules_from_bzl will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (no JDK)](:ubuntu: 18.04 (no JDK))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Tracking issue: Rule usage metrics

We should gather download counts of rules by version by day. This should be provided directly to rule owners so they can see adoption and migration trends. The rate of user change to newer versions informs the choice of how much effort to devote to backwards compatibility.

rename 'master' branch to 'main'

Wrong download url in release notes of 0.0.1

The WORKSPACE snipped in the release notes of 0.0.1 references an url that returns 404: https://github.com/bazelbuild/bazel_federation/releases/download/0.0.1/bazel_federation-0.0.1.tar.gz

The correct url should be https://github.com/bazelbuild/bazel-federation/releases/download/0.0.1/bazel_federation-0.0.1.tar.gz (bazelbuild/bazel-federation vs bazelbuild/bazel_federation).

Flag --incompatible_disable_depset_items will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_disable_depset_items will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (no JDK)](:ubuntu: 18.04 (no JDK))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Flag --incompatible_no_implicit_file_export will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_no_implicit_file_export will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (no JDK)](:ubuntu: 18.04 (no JDK))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

remote_jdk12 repositories should have a sha256 attribute

@remote_jdk12_linux, @remote_jdk12_macos and @remote_jdk12_windows do not have sha256 attributes. For example:

bazel-federation/java_repositories.bzl

Lines 263 to 272 in 9d51bfe

    
           def remote_jdk12_repos(): 
        
               maybe( 
        
                   http_archive, 
        
                   name = "remote_jdk12_linux", 
        
                   build_file_content = "java_runtime(name = 'runtime', srcs =  glob(['**']), visibility = ['//visibility:public'])", 
        
                   strip_prefix = "zulu12.2.3-ca-jdk12.0.1-linux_x64", 
        
                   urls = [ 
        
                       "https://mirror.bazel.build/openjdk/azul-zulu12.2.3-ca-jdk12.0.1/zulu12.2.3-ca-jdk12.0.1-linux_x64.tar.gz", 
        
                   ], 
        
               )

As stated in the http_archive documentation

It is a security risk to omit the SHA-256 as remote files can change. At best omitting this field will make your build non-hermetic. It is optional to make development easier but should be set before shipping.

Flag --incompatible_load_cc_rules_from_bzl will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_load_cc_rules_from_bzl will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 18.04 (no JDK)](:ubuntu: 18.04 (no JDK))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Bazel Federation failing on buildkite (bazel_skylib)

Bazel federation is failing on buildkite for bazel@HEAD. Particularly, a bazel_skylib test is failing. It's tied to some CI changes (and not Bazel changes)

Example failure: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1609#ce13c59d-d31e-47e9-a9e9-1a0dbacd2616

This was fixed in bazel_skylib with bazelbuild/bazel-skylib@68acaa5

Bazel Federation failing in Bazel downstream pipeline

Due to rules_cc breakage caused by the flip of --incompatible_require_linker_input_cc_api

An attempt to upgrade rules_cc failed: #125

/cc @aiuto @oquenchil

Flag --incompatible_disallow_empty_glob will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_disallow_empty_glob will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Flag --incompatible_no_implicit_file_export will break Bazel Federation in a future Bazel release

Incompatible flag --incompatible_no_implicit_file_export will be enabled by default in a future Bazel release [1], thus breaking Bazel Federation.

The flag is documented here: bazelbuild/bazel#10225

Please check the following CI builds for build and test results:

Never heard of incompatible flags before? We have documentation that explains everything.

If you don't want to receive any future issues for Bazel Federation or if you have any questions,
please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

[1] The target release hasn't been determined yet. Our tool will update the issue title once the flag flip has been scheduled.

Add rules_proto to federation

It would be great if rules_proto is part of the federation. It's a dependency of Protobuf (starting with 3.9.1), and also soon used by most projects.

I tried to add it to repositories.bzl but failed because there is a, unfortunately unavoidable, dependency cycle between rules_proto and Protobuf. Is there any existing mechanism to resolve this issue?

Flag --incompatible_load_java_rules_from_bzl will break Bazel Federation in Bazel 1.2.1

Incompatible flag --incompatible_load_java_rules_from_bzl will break Bazel Federation once Bazel 1.2.1 is released.

Please see the following CI builds for more information:

[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:darwin: (OpenJDK 8)](:darwin: (OpenJDK 8))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 18.04 (OpenJDK 11)](:ubuntu: 18.04 (OpenJDK 11))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))
[:windows: (OpenJDK 8)](:windows: (OpenJDK 8))
[:ubuntu: 16.04 (OpenJDK 8)](:ubuntu: 16.04 (OpenJDK 8))

Questions? Please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

Rename rules_python() to rules_python_init()

Maybe it is just me, but names like rules_python() is really confusing.
It leads to the immediate questions:

Am I already adding the rules?
Why do I have to call rules_python_setup() afterwards?

My feeble little mind would be way more on board if it would be called: rules_python_init

Bazel Federation (rules_python) fail on BuildKite

https://buildkite.com/bazel/bazel-federation/builds/435#5f705d2f-3523-49e9-bf73-b296a6279af6

@rules_python//examples/boto:boto_test                                   FAILED in 3 out of 3 in 0.7s

Test log:

Traceback (most recent call last):
  File "/private/var/tmp/_bazel_buildkite/93edee54d3d38d81a5ece387f62d48b8/sandbox/darwin-sandbox/23/execroot/rules_python_federation_example/bazel-out/darwin-fastbuild/bin/external/rules_python/examples/boto/boto_test.runfiles/rules_python/examples/boto/boto_test.py", line 16, in <module>
    import unittest
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/__init__.py", line 60, in <module>
    from .async_case import IsolatedAsyncioTestCase
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/unittest/async_case.py", line 1, in <module>
    import asyncio
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/__init__.py", line 8, in <module>
    from .base_events import *
  File "/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py", line 18, in <module>
    import concurrent.futures
  File "/private/var/tmp/_bazel_buildkite/93edee54d3d38d81a5ece387f62d48b8/sandbox/darwin-sandbox/23/execroot/rules_python_federation_example/bazel-out/darwin-fastbuild/bin/external/rules_python/examples/boto/boto_test.runfiles/examples_boto_pypi__futures_3_3_0/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/private/var/tmp/_bazel_buildkite/93edee54d3d38d81a5ece387f62d48b8/sandbox/darwin-sandbox/23/execroot/rules_python_federation_example/bazel-out/darwin-fastbuild/bin/external/rules_python/examples/boto/boto_test.runfiles/examples_boto_pypi__futures_3_3_0/concurrent/futures/_base.py", line 414
    raise exception_type, self._exception, self._traceback

This only happens on macOS.

Support both distro-only and full-repo testing of member projects

Currently each member project has a single set of tests that are being executed as part of any presubmit on Buildkite. These test targets are hard-coded in a project-specific yaml file, and they
are almost identical to the "regular" presubmit configuration of each individual project, covering all files in the repository. This has been working so far since repositories.bzl imports member repositories in their entirety.

However, we want the federation to reference dedicated releases of its member projects, which only contain the files that end-users actually need. As a result, we would need to remove all tests that cover files that are not part of such releases.

On the other hand, doing so would introduce a new problem: A passing presubmit would only mean that the releases of rules_a and rules_b are compatible, even though rules_b could have broken rules_a internal tests.

As a result, I propose to split the presubmit configuration for each member project into two configurations, rules_foo_release.yml and rules_foo_internal.yml (names TBD).
If someone proposed a new rules_foo version, CI sould do two things:

Test the releases of all other projects by running all targets in the rules_*_release.yml files
Test the entire repositories of all other projects (at the commit of their release) by running all targets in the rules_*_internal.yml files.

Example: #74

Add pipeline for pre-release testing of projects

Scenario: Someone has merged a new feature into rules_foo and thus wants to a) cut a new release and b) add it to the federation. Currently the process looks like this:

Bump rules_foo version number
Build rules_foo release artifacts
Create a new rules_foo release on GitHub
Sent a PR to the federation repo to update repositories.bzl with the new version

Unfortunately we had several cases where the release distribution was buggy (e.g. missing files), which was only discovered in step 3 when the federation presubmit pipeline failed. As a result, the created and published rules_foo release was basically worthless since a patch release had to be cut.

Ideally we'd have an additional presubmit pipeline that could vet potential rule releases before they're actually being cut.

Changes to docs/... should not trigger the pipeline.

Those files are not processed. They go right to https://bazelbuild.github.io/bazel-federation/

Flag --incompatible_disable_depset_items will break Bazel Federation in a future Bazel release

Incompatible flag --incompatible_disable_depset_items will be enabled by default in a future Bazel release [1], thus breaking Bazel Federation.

The flag is documented here: bazelbuild/bazel#9017

Please check the following CI builds for build and test results:

Never heard of incompatible flags before? We have documentation that explains everything.

If you don't want to receive any future issues for Bazel Federation or if you have any questions,
please file an issue in https://github.com/bazelbuild/continuous-integration

Important: Please do NOT modify the issue title since that might break our tools.

[1] The target release hasn't been determined yet. Our tool will update the issue title once the flag flip has been scheduled.

	def remote_jdk12_repos():
	maybe(
	http_archive,
	name = "remote_jdk12_linux",
	build_file_content = "java_runtime(name = 'runtime', srcs = glob(['**']), visibility = ['//visibility:public'])",
	strip_prefix = "zulu12.2.3-ca-jdk12.0.1-linux_x64",
	urls = [
	"https://mirror.bazel.build/openjdk/azul-zulu12.2.3-ca-jdk12.0.1/zulu12.2.3-ca-jdk12.0.1-linux_x64.tar.gz",
	],
	)

bazelbuild / bazel-federation Goto Github PK

bazel-federation's People

Contributors

Stargazers

Watchers

Forkers

bazel-federation's Issues

Recommend Projects

Recommend Topics

Recommend Org