please-build / python-rules Goto Github PK
View Code? Open in Web Editor NEWPython rules for the Please build system
License: Apache License 2.0
Python rules for the Please build system
License: Apache License 2.0
Hi, I can't build a Python pex
file on FreeBSD because pip is unable to compile the pynacl
library (in fact the libsodium
dependency). The error can be reproduced quite simply:
vagrant@freebsd:~ % ls
libsodium-1.0.18-stable.tar.gz
vagrant@freebsd:~ % tar -xf libsodium-1.0.18-stable.tar.gz
vagrant@freebsd:~ % ls
libsodium-1.0.18-stable.tar.gz libsodium-stable
vagrant@freebsd:~ % mv libsodium-stable libsodium#stable
vagrant@freebsd:~ % cd libsodium#stable/
vagrant@freebsd:~/libsodium#stable % ./con
configure* contrib/
vagrant@freebsd:~/libsodium#stable % ./configure
checking build system type... x86_64-unknown-freebsd12.2
checking host system type... x86_64-unknown-freebsd12.2
checking target system type... x86_64-unknown-freebsd12.2
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... configure: error: unsafe absolute working directory name
vagrant@freebsd:~/libsodium#stable %
(using FreeBSD12)
Exactly the same error occurs in the Please logs, eg:
× Building wheel for pynacl (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [105 lines of output]
/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
!!
********************************************************************************
The license_file parameter is deprecated, use license_files instead.
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
********************************************************************************
!!
parsed = self.parsers.get(option_name, lambda x: x)(value)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.freebsd-12.3-STABLE-amd64-cpython-39
creating build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/__init__.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/encoding.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/exceptions.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/hash.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/hashlib.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/public.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/secret.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/signing.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
copying src/nacl/utils.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
creating build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/pwhash
copying src/nacl/pwhash/__init__.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/pwhash
copying src/nacl/pwhash/_argon2.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/pwhash
copying src/nacl/pwhash/argon2i.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/pwhash
copying src/nacl/pwhash/argon2id.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/pwhash
copying src/nacl/pwhash/scrypt.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/pwhash
creating build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/__init__.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_aead.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_box.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_core.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_generichash.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_hash.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_kx.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_pwhash.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_scalarmult.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_secretbox.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_secretstream.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_shorthash.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/crypto_sign.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/randombytes.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/sodium_core.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/bindings/utils.py -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl/bindings
copying src/nacl/py.typed -> build/lib.freebsd-12.3-STABLE-amd64-cpython-39/nacl
running build_clib
checking build system type... x86_64-unknown-freebsd12.3
checking host system type... x86_64-unknown-freebsd12.3
checking target system type... x86_64-unknown-freebsd12.3
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... configure: error: unsafe absolute working directory name
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
main()
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
json_out['return_val'] = hook(**hook_input['kwargs'])
File "/usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 261, in build_wheel
return _build_backend().build_wheel(wheel_directory, config_settings,
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 416, in build_wheel
return self._build_with_temp_dir(['bdist_wheel'], '.whl',
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 401, in _build_with_temp_dir
self.run_setup()
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 338, in run_setup
exec(code, locals())
File "<string>", line 202, in <module>
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 107, in setup
return distutils.core.setup(**attrs)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 346, in run
self.run_command("build")
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1234, in run_command
super().run_command(command)
File "/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-build-env-_zdguq2n/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "<string>", line 160, in run
File "/usr/local/lib/python3.9/subprocess.py", line 373, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-install-_5cuv1uf/pynacl_bf5fea3585644001b89db417bbde9a23/src/libsodium/configure', '--disable-shared', '--enable-static', '--disable-debug', '--disable-dependency-tracking', '--with-pic', '--prefix', '/home/vagrant/builds/xxx/plz-out/tmp/libs/_pynacl#srcs._build/pip-install-_5cuv1uf/pynacl_bf5fea3585644001b89db417bbde9a23/build/temp.freebsd-12.3-STABLE-amd64-cpython-39']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pynacl
Failed to build pynacl
ERROR: Could not build wheels for pynacl, which is required to install pyproject.toml-based projects
Migrate over the behave, pytest and custom runner tests once thought-machine/please#2081 is done
Hello, I've hit an issue using please in GitHub actions. The plugin doesn't seem to use the Python version that is set up by the 'setup-python' action. It uses the version installed by default for the operating system instead.
This is true even for this repository itself. I have created a fork and made the following changes:
import sys
def test_python_version_is_expected():
assert sys.version_info[0] == 3
assert sys.version_info[1] == 12
This test fails for the 3.12 version as you can see in the image below:
It seems like the tests all use 3.10 rather than the versions setup by the setup-action.
It's expected that @self
as a subrepo reference (e.g. @self//third_party/python:foo
) when inside a plugin build file should evaluate to the plugin itself, but currently the reference is evaluated in whatever subrepo the build file is being subincluded from, resulting in directories not being found etc.
Hi !
I'm building a small CLI tool and I'm using please to produce a .pex file that I want to distribute to my user.
My problem is that if I distribute a newer version of the .pex file, it appears to get the same stamp, and will try to extract to the same cache folder. Since the pex has already been extracted here, it's not extracted again, and the old code runs.
If my understanding is correct, the __PEX_STAMP__
value, which is used to select the path where the pex file is extracted comes from $RULE_HASH
which doesn't change when the built content changes, but only when the rules themselves changes.
I'm not sure I really understand why the rule hash is used instead of the content of the built files themselves. Since I'm not building for a container use case, I need to be able to update the app of my user's boxes.
Would you accept a PR allowing to use the hash of the files instead of the rule hash ? Would that be a fix ?
With python.test_runner = "pytest"
and python.testrunner_deps = "///python//third_party/python:pytest_bootstrap"
, plz cover
on a python_test
fails with the following error:
bash-4.4$ $TEST
Traceback (most recent call last):
File "[...]/python/python3.10/python3.10/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "[...]/python/python3.10/python3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/tmp/plz_sandbox/[...]/__main__.py", line 488, in <module>
File "/tmp/plz_sandbox/[...]/__main__.py", line 472, in run
File "/tmp/plz_sandbox/[...]/__main__.py", line 409, in main
File "/tmp/plz_sandbox/[...]/__main__.py", line 382, in initialise_coverage
ModuleNotFoundError: No module named 'coverage'
This appears to be because coverage isn't part of the built-in pytest test runner, nor are several other third-party packages required by plz cover
.
I came across this when defining an entry point for Flake8 (this isn't quite a MWE because it's missing a lot of dependencies, but you get the gist):
## third_party/python3/BUILD
python_wheel(
name = "flake8",
outs = [
"flake8",
"flake8-6.0.0.dist-info",
],
entry_points = "flake8.main.cli:main",
version = "6.0.0",
zip_safe = False,
deps = [
# ...
],
)
Running flake8 --version
via the generated pex works fine:
$ plz run --env //third_party/python3:flake8 -- --version
6.0.0 (flake8-alfred: 1.1.1, flake8-bugbear: 23.7.10, flake8-builtins: 2.1.0, flake8-tuple: 0.4.1, mccabe: 0.7.0, pycodestyle: 2.10.0, pyflakes: 3.0.1) CPython 3.9.16 on Linux
but running Flake8 on a Python source file doesn't:
$ plz run --env //third_party/python3:flake8 -- test.py
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 158, in _get_module_details
code = loader.get_code(mod_name)
File "[...]/plz-out/bin/third_party/python3/flake8.pex/__main__.py", line 249, in get_code
return module.__loader__.get_code(fullname)
File "<frozen importlib._bootstrap_external>", line 923, in get_code
File "<frozen importlib._bootstrap_external>", line 527, in _check_name_wrapper
ImportError: loader for __flake8_main__ cannot handle third_party.python3.__flake8_main__
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "[...]/plz-out/bin/third_party/python3/flake8.pex/__main__.py", line 397, in <module>
result = run(explode)
File "[...]/plz-out/bin/third_party/python3/flake8.pex/__main__.py", line 377, in run
return main()
File "[...]/plz-out/bin/third_party/python3/flake8.pex/__main__.py", line 371, in main
runpy.run_module(ENTRY_POINT, run_name='__main__')
File "/usr/lib/python3.9/runpy.py", line 221, in run_module
mod_name, mod_spec, code = _get_module_details(mod_name)
File "/usr/lib/python3.9/runpy.py", line 160, in _get_module_details
raise error(format(e)) from e
ImportError: loader for __flake8_main__ cannot handle third_party.python3.__flake8_main__
test.py:1:1: F401 'sys' imported but unused
(Note the final line: it appears Flake8 does actually run, but something goes wrong in please_pex afterwards.)
From discussions with Tatskaari, it seems a good way to go would be to allow some tags to be specified in the config for a given architecture, e.g. ["any", "macosx-10.6-intel-cp36", ...]
for darwin, which would be passed to the tool. The tool would then look for tags from whatever index that match the given architecture.
This issue was orignally raised here:
thought-machine/please#2304
A fix has been implemented here:
thought-machine/please#2304
This needs to be ported.
E.g. it's unable to find any wheels for version 3.12.2 of protobuf
, even though it looks on PyPI like there are wheels available.
python_wheel(
name = "protobuf",
outs = ["google"],
- version = "3.12.2",
+ version = "3.18.1",
deps = [":six"],
)
This is regarding pex executables generated from the tools/please_pex/pex/pex_main.py template.
If PEX_NOCACHE=true
, and PEX_EXPLODE=1
or ZIP_SAFE=True
, then the pex gets unzipped to a temp directory. The explode_zip()
function returns a context manager which is supposed to clean up the temp directory after the main entry point exits. The problem is the cleanup doesn't happen if there is an exception thrown in the entry point code. Here's the relevant code from the _explode_zip()
context manager:
yield
if no_cache:
import shutil
shutil.rmtree(basepath)
Since there's no try/finally
around the yield
statement, if the entry point script throws an exception, then the code after the yield
to cleanup the cache doesn't get run. Note that this is true also if the entry point script exits using sys.exit
, even if it exits with sys.exit(0)
, since that works by raising a SystemExit
exception.
I could see not deleting the tmp directory after an exception being a useful feature to help a developer debug the failure, but only if there was some message printed to the console informing the user where the tmp directory is. I was only able to figure out what was going on by looking through the please source code. I had a case where I was using numpy and pandas, and the exploded dependencies took up 390MB. As you can imagine, after iteratively testing of my code for a while, there were GBs of space being used by exploded tmp directories.
Solutions.
I think the best solution would be to make it so the delete happens even if an exception is thrown. So change the code in the context manager to something like:
try:
yield
finally:
if no_cache:
import shutil
shutil.rmtree(basepath)
You could add an additional environment variable or flag to make it not delete the cache if desired.
Alternatively, at least add some documentation to the python_binary()
docs stating that the entry point script passed as the main
parameter is expected not to call sys.exit
or let an exception escape. Though I think that would be a bad solution, because there needs to be some way to write python executables that exit with a non-zero exit status.
Error building target //third_party/python:_protobuf#download: exit status 1
CRITICAL:root:Couldn't find any matching urls in the index
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/jpoole/proto-rules/plz-out/bin/python/tools/wheel_resolver/wheel_resolver.pex/__main__.py", line 390, in <module>
File "/home/jpoole/proto-rules/plz-out/bin/python/tools/wheel_resolver/wheel_resolver.pex/__main__.py", line 377, in run
File "/home/jpoole/proto-rules/plz-out/bin/python/tools/wheel_resolver/wheel_resolver.pex/__main__.py", line 367, in main
File "/usr/lib/python3.6/runpy.py", line 208, in run_module
return _run_code(code, {}, init_globals, run_name, mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "tools/wheel_resolver/src/resolve.py", line 60, in <module>
File "tools/wheel_resolver/src/resolve.py", line 51, in main
File "tools/wheel_resolver/src/wheel_tags/tag
We should catch this and return something more useful
The version of pip in Debian 12's python3-pip
package removes the --system
option, so pip_library
no longer works:
$ plz build //third_party/python:colorama
Build stopped after 490ms. 1 target failed:
//third_party/python:_colorama#srcs
Error building target //third_party/python:_colorama#srcs: exit status 2
Usage:
/usr/bin/python3 -m pip install [options] <requirement specifier> [package-index-options] ...
/usr/bin/python3 -m pip install [options] -r <requirements file> [package-index-options] ...
/usr/bin/python3 -m pip install [options] [-e] <vcs project url> ...
/usr/bin/python3 -m pip install [options] [-e] <local project path> ...
/usr/bin/python3 -m pip install [options] <archive url/path> ...
no such option: --system
We should update the readme here to include all the config options and what they do like I've done with https://github.com/please-build/java-rules
We should also write codelabs for all of these plugins
We should move the wheel downloader's python_binary()
to //tools/wheel_downloader/src
and make //tools/wheel_downloader
a remote file that downloads from the releases page.
The built-in test runner pexes generated by please_pex
currently don't all work (pytest in particular is quite broken), even when --add_test_runner_deps
is passed to please_pex
by python_test
. This appears to be because the src
s in the test runner modules in //third_party/python
, the src
s in the //third_party/python:*_bootstrap
filegroups, and the testRunnerDeps
logic in tools/please_pex/pex/pex.go
's SetTest
function have gradually fallen out of sync with each other, so only a subset of the necessary third-party dependencies are actually making it into the .bootstrap
directory.
If you have a python_wheel
in the top-level BUILD file, the #wheel
hidden target errors out because $PKG
is empty. We should probably remove the --prefix
flag from the build command if this is the case.
pip_library help says it takes outs
as an argument, but it doesn't.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.