benley/bazel_rules_pex

Python PEX rules for Bazel

bazel_rules_pex's Issues

Use upstream pex without modifications

Blocked on pex-tool/pex#277, same as apache/incubator-heron#881

cache issue

Hi, Ben

while you are working on Bazel's pex rules, I'm wondering if you have seen the following error message arising from pex:

    sys.exit(main())
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/_pex.py", line 159, in main
    resolver_options_builder, interpreter=interpreter)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/bin/pex.py", line 472, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 191, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/travis/.cache/bazel/_bazel_travis/030a6cc9b27b34f2fe0e137b942f59bf/heron/bazel-out/local_linux-fastbuild/bin/third_party/pex/_pex.runfiles/__main__/third_party/pex/pex/resolver.py", line 254, in build
    os.rename(target + '~', target)
OSError: [Errno 2] No such file or directory

I thought apache/incubator-heron#1052 should solve apache/incubator-heron#1051, but it didn't.

Any idea?

Add tests

py_library imports ignored when linking into pex_binary

TL;DR: If I have a py_library(..., srcs=["foo.py"], imports=["."]), then with the standard Bazel Python rules I can use import foo to reference it. If I link this py_library into a pex_binary, it ignores the imports part, so the files get packed into the pex as full/workspace/path/foo.py, instead of as foo.py in the root of the pex.

Expected result: I should be able to change standard Bazel py_binary into a pex_binary and have it magically work. This probably means shuffling files around, or packaging each py_library into a wheel, and packaging those together into the pex?

Additional details

I have a Python mono-repo that is set up something like this:

• package_foo/setup.py - A "standard" setuptools setup.py to build a wheel
• package_foo/foo/__init__.py - Python package foo
• somebinary/bin.py - A Python program that uses import foo

(Imagine more packages and binaries)

Each subdirectory is a "standard" Python package. We are still evaluating the various monorepo build tools (Pants, Buck, Bazel), so this lets us make it work fairly easily. I'd like to just "drop in" the appropriate BUILD rules, but I can't figure out how to make pex_binary work in this case, since I need to "fix" the import path somehow.

PS. Thanks for creating these rules! Being able to directly link wheels is great.

pex_binary fails on jsonschema requirement with python 3 target

When targeting python 3.5, requiring jsonschema leads to pex trying to install functools32 (and failing, since the package is python 2.7 only).

$ /usr/local/bin/python3 --version
Python 3.5.2
$ bazel help | head -1
                                                  [bazel release 0.3.0-homebrew]
$ mkdir repro
$ cd repro
$ cat > repro.py <<EOF
import jsonschema

if __name__ == '__main__':
   print('hello')
EOF
$ cat > BUILD <<EOF
pex_binary(
    name="repro",
    srcs=['repro.py'],
    main="repro.py",
    interpreter="/Usr/local/bin/python3",
    reqs=[
        "jsonschema==2.5.1"
    ]
 )
EOF
$ cd ..
$ bagel run //repro
...
**** Failed to install functools32-3.2.3-2. stderr:
This backport is for Python 2.7 only.
...

jsonschema declares its dependency as:

Requires-Dist: functools32; python_version=='2.7'

It looks like pex is not reading this information correctly.

This is with bazel_rules_pex 4882f2d.

Need way to specify target platform

Hi!

I am using your rules, and rules are awesome.
I have single small problem: on OSX I am building PEX binary and then put it to centos-based docker image.

As result, PEX image does not work because it includes OSX-based numpy/other libraries with C modules.

It would be great to provide some way to build PEX-binaries for target platform (ideally - handle "--cpu" flag from bazel, which used for cross-compile)

Thank you!

pex_wrapper.pex failed when I run the first invocation - second invocation works successfully

 bazel build heron/...
INFO: Found 2 targets...
ERROR: /Users/karthikz/workspace/connectors/heron/spouts/pulsar/src/python/BUILD:12:1: null failed: pex_wrapper.pex failed: error executing command bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex --pex-root .pex --entry-point heron.spouts.pulsar.src.python.pulsar_spout --output-file ... (remaining 4 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 326, in execute
  File ".bootstrap/_pex/pex.py", line 258, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 290, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 369, in _execute
  File ".bootstrap/_pex/pex.py", line 427, in execute_entry
  File ".bootstrap/_pex/pex.py", line 432, in execute_module
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 180, in run_module
    fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/private/var/tmp/_bazel_karthikz/fa631bb0f7ae8f36398ceb2660e4c283/execroot/connectors/.pex/install/pex_wrapper-0.1-py2-none-any.whl.bce01df26c502f8e244b96dc2f1e4682854a40a0/pex_wrapper-0.1-py2-none-any.whl/pex_wrapper.py", line 109, in <module>
    main()
  File "/private/var/tmp/_bazel_karthikz/fa631bb0f7ae8f36398ceb2660e4c283/execroot/connectors/.pex/install/pex_wrapper-0.1-py2-none-any.whl.bce01df26c502f8e244b96dc2f1e4682854a40a0/pex_wrapper-0.1-py2-none-any.whl/pex_wrapper.py", line 63, in main
    resolver_options_builder)
  File "/private/var/tmp/_bazel_karthikz/fa631bb0f7ae8f36398ceb2660e4c283/execroot/connectors/.pex/install/pex-1.1.14-py2.py3-none-any.whl.8350b5e3ea62522c62fc3adc036502afbb66b0d3/pex-1.1.14-py2.py3-none-any.whl/pex/bin/pex.py", line 489, in build_pex
    resolveds = resolver.resolve(resolvables)
  File "/private/var/tmp/_bazel_karthikz/fa631bb0f7ae8f36398ceb2660e4c283/execroot/connectors/.pex/install/pex-1.1.14-py2.py3-none-any.whl.8350b5e3ea62522c62fc3adc036502afbb66b0d3/pex-1.1.14-py2.py3-none-any.whl/pex/resolver.py", line 200, in resolve
    dist = self.build(package, resolvable.options)
  File "/private/var/tmp/_bazel_karthikz/fa631bb0f7ae8f36398ceb2660e4c283/execroot/connectors/.pex/install/pex-1.1.14-py2.py3-none-any.whl.8350b5e3ea62522c62fc3adc036502afbb66b0d3/pex-1.1.14-py2.py3-none-any.whl/pex/resolver.py", line 263, in build
    os.rename(target + '~', target)
OSError: [Errno 2] No such file or directory
INFO: Elapsed time: 18.596s, Critical Path: 13.09s

name 'set' is not defined

I get the following error when pex_repositories() is called:

# bazel build ...
ERROR: /root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/external/io_bazel_rules_pex/pex/pex_rules.bzl:60:18: name 'set' is not defined
ERROR: error loading package '': Extension 'pex/pex_rules.bzl' has errors
ERROR: error loading package '': Extension 'pex/pex_rules.bzl' has errors
INFO: Elapsed time: 0.917s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

Environment

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

Bazel Version

# bazel version
Build label: 0.15.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jul 17 12:22:40 2018 (1531830160)
Build timestamp: 1531830160
Build timestamp as int: 1531830160

WORKSPACE

git_repository(
    name = "io_bazel_rules_pex",
    remote = "https://github.com/benley/bazel_rules_pex.git",
    tag = "0.3.0",
)
load("@io_bazel_rules_pex//pex:pex_rules.bzl", "pex_repositories")
pex_repositories()

Bazel 0.10 fails with Extension file not found

I'm on macos trying to pull in bazel_rules_pex. It was working before 0.10, but after upgrading, I get an error like:

ERROR: /Users/flaub/src/vertex/work1-imac/private/vision/web/BUILD:59:1: every rule of type pex_binary implicitly depends upon the target '@io_bazel_rules_pex//pex:pex_wrapper', but this target could not be found because of: error loading package '@io_bazel_rules_pex//pex': Extension file not found. Unable to load package for '//pex:pex_rules.bzl': BUILD file not found on package path
ERROR: Analysis of target '//vision/web:demo.pex' failed; build aborted: error loading package '@io_bazel_rules_pex//pex': Extension file not found. Unable to load package for '//pex:pex_rules.bzl': BUILD file not found on package path

My WORKSPACE has this in it:

http_archive(
    name = "io_bazel_rules_pex",
    sha256 = "4e068ab38589b2baf22d2c6c58b7cb3004d14191bc4f9fdb0baed4275520943f",
    strip_prefix = "bazel_rules_pex-e2746c5c4ed70f7a56b724eec733a3fb32239b77",
    url = "https://github.com/benley/bazel_rules_pex/archive/e2746c5c4ed70f7a56b724eec733a3fb32239b77.zip",
)

load("@io_bazel_rules_pex//pex:pex_rules.bzl", "pex_repositories")

pex_repositories()

After some digging, it appears that the bazel_prelude defined in the io_bazel_rules_pex workspace is somehow effecting my workspace. So I removed the prelude from io_bazel_rules_pex and now it seems to work.

accept python sdists as deps somehow

the eggs attribute works with eggs and wheels, but currently there is no easy way to build a distribution from a python sdist archive.

One of the simplest ways to accomplish this would be to add a find_links attribute that added paths to be searched for requirements.

What might be better is a new rule type (or macro) that converted sdists into wheels, which could then be put into a pex_library. This would result in better build caching.

setting interpreter attribute isn't enough

Summary

This is similar to the bug here pantsbuild/pants#4490, sadly setting interpreter attribute in Pex's rules to python3.6 isn't enough, but you also you need to have /usr/bin/env python and pip pointing to Python 3.6 in the host machine.

Open Questions

I was wondering, although I am not sure if this is the right place to ask, why is python rules not using sandbox like example Golang, there you can specify the go version in the WORKSPACE and Bazel downloads the specific Golang environment in a sandbox and use this later, instead of relying on host system, in contrast, python rules rely on the host system which makes setting up working with Bazel and Python too cumbersome.

Old Version of Wheel being Sourced

Broken with bazel 0.3.2 sandboxing

See https://travis-ci.org/benley/bazel_rules_pex/jobs/168495782

ERROR: /home/travis/build/benley/bazel_rules_pex/examples/BUILD:1:1: null failed: linux-sandbox failed: error executing command /home/travis/.cache/bazel/_bazel_travis/a596ea0b837be742c3e08e7b27f0d368/execroot/bazel_rules_pex/_bin/linux-sandbox ... (remaining 12 argument(s) skipped).
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 320, in execute
  File ".bootstrap/_pex/pex.py", line 78, in _activate
  File ".bootstrap/_pex/environment.py", line 132, in activate
  File ".bootstrap/_pex/environment.py", line 176, in _activate
  File ".bootstrap/_pex/environment.py", line 121, in update_candidate_distributions
  File ".bootstrap/_pex/environment.py", line 107, in load_internal_cache
  File ".bootstrap/_pex/environment.py", line 95, in write_zipped_internal_cache
  File ".bootstrap/_pex/util.py", line 176, in cache_distribution
  File ".bootstrap/_pex/common.py", line 124, in safe_open
  File ".bootstrap/_pex/common.py", line 112, in safe_mkdir
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nonexistent'

Add examples

Surely there should be some simple usage examples in this repo.

req_files only work when files is included in data attributes

Tried this

filegroup(
    name = "requirements",
    srcs = [
        "requirements.txt",
    ],
)

pex_pytest(
    name = "default_test",
    srcs = glob(["**/*.py"]),
    deps = [
        ":default_library",
    ],
    req_files = [
       ":requirements",
    ],
)

And it fails with

  File "/Users/mouad/.cache/bazel/_bazel_mouad/bazel-sandbox/8258116086626091877/execroot/__main__/.pex/install/pex_wrapper-0.1-py2-none-any.whl.213146efb1c3b3423224a6d8dabeba348a129f7e/pex_wrapper-0.1-py2-none-any.whl/pex_wrapper.py", line 131, in <module>
    main()
  File "/Users/mouad/.cache/bazel/_bazel_mouad/bazel-sandbox/8258116086626091877/execroot/__main__/.pex/install/pex_wrapper-0.1-py2-none-any.whl.213146efb1c3b3423224a6d8dabeba348a129f7e/pex_wrapper-0.1-py2-none-any.whl/pex_wrapper.py", line 85, in main
    resolver_options_builder)
  File "/Users/mouad/.cache/bazel/_bazel_mouad/bazel-sandbox/8258116086626091877/execroot/__main__/.pex/install/pex-1.2.9-py2.py3-none-any.whl.5c6ee6196c29e6a1d0cc5e5b4d54bb3369c2b93f/pex-1.2.9-py2.py3-none-any.whl/pex/bin/pex.py", line 528, in build_pex
    resolvables.extend(requirements_from_file(requirements_txt, resolver_option_builder))
  File "/Users/mouad/.cache/bazel/_bazel_mouad/bazel-sandbox/8258116086626091877/execroot/__main__/.pex/install/pex-1.2.9-py2.py3-none-any.whl.5c6ee6196c29e6a1d0cc5e5b4d54bb3369c2b93f/pex-1.2.9-py2.py3-none-any.whl/pex/requirements.py", line 97, in requirements_from_file
    with open(filename, 'r') as fp:
IOError: [Errno 2] No such file or directory: 'services/delivery/requirements.txt'

After a bit of debugging, I found out that my requirements.txt is not actually in the folder /Users/mouad/.cache/bazel/_bazel_mouad/execroot/__main__/services/delivery, but then when I changed the target to:

pex_pytest(
    name = "default_test",
    srcs = glob(["**/*.py"]),
    deps = [
        ":default_library",
    ],
    req_files = [
       ":requirements",
    ],
    data = [
       ":requirements",
    ],
)

It worked!

Is this by design or I am missing something?

For a script that depends on google-cloud re-builds are taking 70s

I don't know if the pex rule is re-downloading it every time or what but it makes development pretty impossible. Is this expected? Shouldn't it be cached in some way?

here's what my rule looks like:

pex_binary(
    name = "update-dns",
    main = "update_dns.py",
    reqs = [
        "enum34==1.1.6",
        "google-cloud==0.26.0"
    ],
    zip_safe = False,
    deps = [":update-dns-lib"],
)

py_library(
    name = "update-dns-lib",
    srcs = glob(["update_dns.py"]),
)

If I make a change to update_dns.py and then re-run the script with bazel it takes 70 seconds or so. Reruns without any changes to the file are quick.

Could you please add a license to your project.

Could you please add a license to your project (MIT if possible) so that I can submit my PR that considers imports?

Old Version of Wheel being Sourced

I'm seeing the following error:

File "/home/.cache/bazel/_bazel_zanes/ea59e07677eec0e868edc628a8df6a9d/execroot/repo/.pex/interpreters/CPython-3.4.3/wheel-0.25.0-py3.4.egg/wheel/bdist_wheel.py", line 229, in run
File "/home/.cache/bazel/_bazel_zanes/ea59e07677eec0e868edc628a8df6a9d/execroot/repo/.pex/interpreters/CPython-3.4.3/wheel-0.25.0-py3.4.egg/wheel/bdist_wheel.py", line 394, in egg2dist
File "/home/.cache/bazel/_bazel_zanes/ea59e07677eec0e868edc628a8df6a9d/execroot/repo/.pex/interpreters/CPython-3.4.3/wheel-0.25.0-py3.4.egg/wheel/metadata.py", line 194, in pkginfo_to_dict
TypeError: unorderable types: EntryPoint() < EntryPoint()

This python TypeError was fixed in wheel 0.26.0: https://bitbucket.org/pypa/wheel/issues/148/unorderable-types-error-for-python-3

However, you can see that wheel-0.25.0 is being used. Any idea why this old version of wheel is being used to compile this source?

For reference, here's my pex BUILD file. I'm using python3.4 on Ubuntu 14.04 with Bazel 0.3.1.

pex_binary(
name='app',
main = 'app.py',
srcs= [
'app.py'
],
deps = [
'//app/controllers'
],
interpreter = 'python3.4',
reqs = ['cython', 'numpy', 'pandas'],
eggs = [
'//third_party/python/ppcapp'
],
)

Tag a release + update docs

I had to use commit = "7e6890ff5f806dcc3790b86e4acfed9ad3a82ee7" instead of tag = "0.3.0" in the load(…) as the tag is out of date (from 2016, 28 commits ago) and misses changes for bazel 0.6.0 compatibility.

The docs say to use the tag 0.3.0, so it would be good to update the docs at the same time.

Symlink runfiles tree so that you can do a dev/test iteration without rebuilding PEX

With py_test (PAR-based) rules I can dev/test iterate on an individual test by running the following:

1. bazel test //foo:foo_tests
2. find and update an issue in dependency bar.py
3. run bazel test //foo:foo_tests without rebuilding the PAR because the runfiles are symlinked

The iteration cycle (1 and 2 above) is fast because it essentially lays out the zipfile in the runfiles directory and symlinks files that I may want to edit.

With the PEX rules I cannot do this as the only files that are symlinked are the srcs files from the test itself. This means that I have to rebuild the PEX each time I have to edit a dependent file which makes life quite tedious for any serious development.

What should happen is that the runfiles tree is created similarly to what PAR does.

pex_binary 0.3.0 broken with bazel 0.6

Testing the current release branch from bazel against the existing pex_binary release gives:

$ bazel build :application
ERROR: /Users/hwright/duolingo/duolingo-eta/BUILD:5:1: in pex_binary rule //:application: 
Traceback (most recent call last):
	File "/Users/hwright/duolingo/duolingo-eta/BUILD", line 5
		pex_binary(name = 'application')
	File "/private/var/tmp/_bazel_hwright/084f6c55936b2ad3dbdc77bbb4b384b1/external/io_bazel_rules_pex/pex/pex_rules.bzl", line 142, in _pex_binary_impl
		set(ctx.files.srcs)
The `set` constructor for depsets is deprecated and will be removed. Please use the `depset` constructor instead. You can temporarily enable the deprecated `set` constructor by passing the flag --incompatible_disallow_set_constructor=false.
ERROR: Analysis of target '//:application' failed; build aborted.
INFO: Elapsed time: 25.144s
$

There is, of course, a trivial workaround in the error message, but the rules themselves should be upgraded.

PEX build fails with the following error: bad interpreter: No such file or directory

When I run it with --verbose_failures it gives the following message:

ERROR: /home/vagolikov/.cache/bazel/_bazel_vagolikov/d41052656c7dd24d17b25cebd7e91d28/external/io_bazel_rules_pex/pex/BUILD:4:1: Bootstrapping pex @io_bazel_rules_pex//pex:pex_wrapper failed (Exit 126): bash failed: error executing command 
  (cd /home/vagolikov/.cache/bazel/_bazel_vagolikov/d41052656c7dd24d17b25cebd7e91d28/execroot/__main__ && \
  exec env - \
    LD_LIBRARY_PATH=/data/3rdparty/bpipe/blpapi_cpp_3.10.7.1/Linux:/usr/local/lib \
    PATH=/home/vagolikov/strategy_env/bin:/data/tools/edm/bin:/home/vagolikov/bin:/home/vagolikov/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/vagolikov/bin:/data/tools/edm/bin:/data/tools/eclipse:/data/apps/voltdb/voltdb-community-8.0/bin/ \
  /bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; 
        OUTDIR=$(cd bazel-out/host/bin/external/io_bazel_rules_pex/pex && pwd)
        # Workaround really long shebang lines breaking on linux:
        # Use a /tmp path, but keep the actual venv inside the bazel outdir.
        # Avoids having to worry about cleanup, even if sandboxing is off.
        TMPF=$(mktemp)
        ln -sf "$OUTDIR" "$TMPF"
        VENV="${TMPF}/venv"

        bazel-out/host/bin/external/virtualenv/virtualenv --no-download --quiet --clear "$VENV"
        PYTHON="$VENV/bin/python"

        $VENV/bin/pip install pex             --quiet --no-cache-dir --no-index --build bazel-out/host/bin/external/io_bazel_rules_pex/pex/pexbuild             --find-links $(dirname external/pex_src/file/pex-1.1.14.tar.gz)             --find-links $(dirname external/setuptools_src/file/setuptools-20.10.1.tar.gz)

        # Work around setuptools insistance on writing to the source directory,
        # which is discouraged by Bazel (and annoying)
        cp -r $(dirname external/io_bazel_rules_pex/pex/wrapper/setup.py) bazel-out/host/bin/external/io_bazel_rules_pex/pex/.pex_wrapper

        # Use the bootstrapped pex to build pex_wrapper.pex
        $VENV/bin/pex bazel-out/host/bin/external/io_bazel_rules_pex/pex/.pex_wrapper             --disable-cache --no-index -m pex_wrapper -o bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex             --find-links $(dirname external/pex_src/file/pex-1.1.14.tar.gz)             --find-links $(dirname external/setuptools_src/file/setuptools-20.10.1.tar.gz)             --find-links $(dirname external/requests_src/file/requests-2.11.1.tar.gz)             --find-links $(dirname external/wheel_src/file/wheel-0.29.0.tar.gz)
    ')

se --sandbox_debug to see verbose messages from the sandbox
/bin/bash: /home/vagolikov/.cache/bazel/_bazel_vagolikov/d41052656c7dd24d17b25cebd7e91d28/bazel-sandbox/7332616064763773160/execroot/__main__/tmp/tmp.9sfXNTjQpu/venv/bin/pip: /home/vagolikov/.cache/bazel/_bazel_vagolikov/d41052656c7dd24d17b25cebd7e91d2: bad interpreter: No such file or directory
Target //src/python/word-count-topology:word-count-topology failed to build

Update pex to follow heron

see apache/incubator-heron#1052

28.1 breaks pex?

It may have broken before this version but I was trying out using pex and it seems like its broken. This is me cloning master and seeing if it works.

Starting local Bazel server and connecting to it...
ERROR: /home/tradergt/modules/bazel_rules_pex/pex/pex_rules.bzl:49:18: name 'FileType' is not defined
ERROR: error loading package '': Extension 'pex/pex_rules.bzl' has errors
ERROR: error loading package '': Extension 'pex/pex_rules.bzl' has errors
INFO: Elapsed time: 3.955s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

Virtualenv data glob is too aggressive

I've been noticing that pex rules end up rebuilding unnecessarily. I dug into it and found that building @io_bazel_rules_pex//pex:pex_wrapper builds @virtualenv//:virtualenv. However, immediately after that, building the pex wrapper with --check_up_to_date fails because it claims that the source manifest for virtualenv is out of date. It then stabilizes after building it a second time.

It turns out that running the virtualenv binary sticks a virtualenv.pyc next to virtualenv.py and thus invalidates this aggressive glob.

I fixed it by explicitly excluding that pyc file but I'm hoping that there is a better fix to not use a glob and only include the files that are needed.

Inconsistent Build Error: invalid command 'bdist_wheel'

Some builds will fail with the error below. It doesn't happen consistently and a simple rebuild (no bazel clean) will allow the build to succeed.

see full stacktrace at the bottom

stdout (abbreviation)

# bazel build launch
INFO: Analysed target //launch:launch (0 packages loaded).
INFO: Found 1 target...
ERROR: /Users/dev0/piran/launch/BUILD:11:1: PexPython launch/launch.pex failed (Exit 1)
**** Failed to install MarkupSafe-1.0 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python3.5', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpb8t9gre6']` while trying to execute `['/usr/bin/python3.5', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpb8t9gre6']`",)
):
stdout:

stderr:
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'
...

Environment

# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

Version

# bazel version
Build label: 0.15.2
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jul 17 12:22:40 2018 (1531830160)
Build timestamp: 1531830160
Build timestamp as int: 1531830160

WORKSPACE

git_repository(
    name = "io_bazel_rules_pex",
    remote = "https://github.com/benley/bazel_rules_pex.git",
    commit = "7e6890ff5f806dcc3790b86e4acfed9ad3a82ee7",
)
load("@io_bazel_rules_pex//pex:pex_rules.bzl", "pex_repositories")
pex_repositories()

stdout

# bazel build launch
INFO: Analysed target //launch:launch (0 packages loaded).
INFO: Found 1 target...
ERROR: /Users/dev0/piran/launch/BUILD:11:1: PexPython launch/launch.pex failed (Exit 1)
**** Failed to install MarkupSafe-1.0 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python3.5', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpb8t9gre6']` while trying to execute `['/usr/bin/python3.5', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpb8t9gre6']`",)
):
stdout:

stderr:
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

error: invalid command 'bdist_wheel'


Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex/.bootstrap/_pex/pex.py", line 365, in execute
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex/.bootstrap/_pex/pex.py", line 293, in _wrap_coverage
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex/.bootstrap/_pex/pex.py", line 325, in _wrap_profiling
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex/.bootstrap/_pex/pex.py", line 408, in _execute
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex/.bootstrap/_pex/pex.py", line 466, in execute_entry
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_pex/pex/pex_wrapper.pex/.bootstrap/_pex/pex.py", line 471, in execute_module
  File "/usr/lib/python3.5/runpy.py", line 199, in run_module
    return _run_code(code, {}, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex_wrapper-0.1-py2.py3-none-any.whl.5bc6bd6d5f9c936df9fea2fffef5498fca89699d/pex_wrapper-0.1-py2.py3-none-any.whl/pex_wrapper.py", line 122, in <module>
    main()
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex_wrapper-0.1-py2.py3-none-any.whl.5bc6bd6d5f9c936df9fea2fffef5498fca89699d/pex_wrapper-0.1-py2.py3-none-any.whl/pex_wrapper.py", line 76, in main
    resolver_options_builder)
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex-1.2.13-py2.py3-none-any.whl.6bfeb70d4c4280954ddc331c1f3a49cad35a567d/pex-1.2.13-py2.py3-none-any.whl/pex/bin/pex.py", line 571, in build_pex
    for dist in resolveds:
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex-1.2.13-py2.py3-none-any.whl.6bfeb70d4c4280954ddc331c1f3a49cad35a567d/pex-1.2.13-py2.py3-none-any.whl/pex/resolver.py", line 438, in resolve_multi
    allow_prereleases):
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex-1.2.13-py2.py3-none-any.whl.6bfeb70d4c4280954ddc331c1f3a49cad35a567d/pex-1.2.13-py2.py3-none-any.whl/pex/resolver.py", line 376, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder))
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex-1.2.13-py2.py3-none-any.whl.6bfeb70d4c4280954ddc331c1f3a49cad35a567d/pex-1.2.13-py2.py3-none-any.whl/pex/resolver.py", line 209, in resolve
    dist = self.build(package, resolvable.options)
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex-1.2.13-py2.py3-none-any.whl.6bfeb70d4c4280954ddc331c1f3a49cad35a567d/pex-1.2.13-py2.py3-none-any.whl/pex/resolver.py", line 282, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/install/pex-1.2.13-py2.py3-none-any.whl.6bfeb70d4c4280954ddc331c1f3a49cad35a567d/pex-1.2.13-py2.py3-none-any.whl/pex/resolver.py", line 177, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///root/.cache/bazel/_bazel_root/6f6133ece88806ce1513398e26791cf1/sandbox/processwrapper-sandbox/1/execroot/__main__/.pex/build/MarkupSafe-1.0.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
Target //launch:launch failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 23.637s, Critical Path: 23.09s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

"Relative module names" when importing pex_binary from a bazel git_repository

Here is what I have in my build file:

pex_binary(
name = "example",
srcs = [
    "@tools//start_esp:fetch_service_config.py",
    "@tools//start_esp:start_esp.py",
],
main = "@tools//start_esp:start_esp.py",
reqs = [
    "certifi",
    "mako>=1.0.4",
    "oauth2client>=3.0.0",
    "pyasn1>=0.1.9",
    "pyasn1-modules>=0.0.8",
    "urllib3>=1.16",
],
)

@tools is an externa repository. Trying to run this gives me an error:

Traceback (most recent call last): File ".bootstrap/_pex/pex.py", line 326, in execute File ".bootstrap/_pex/pex.py", line 258, in _wrap_coverage File ".bootstrap/_pex/pex.py", line 290, in _wrap_profiling File ".bootstrap/_pex/pex.py", line 369, in _execute File ".bootstrap/_pex/pex.py", line 427, in execute_entry File ".bootstrap/_pex/pex.py", line 432, in execute_module File "/usr/lib/python2.7/runpy.py", line 170, in run_module mod_name, loader, code, fname = _get_module_details(mod_name) File "/usr/lib/python2.7/runpy.py", line 101, in _get_module_details loader = get_loader(mod_name) File "/usr/lib/python2.7/pkgutil.py", line 464, in get_loader return find_loader(fullname) File "/usr/lib/python2.7/pkgutil.py", line 474, in find_loader for importer in iter_importers(fullname): File "/usr/lib/python2.7/pkgutil.py", line 425, in iter_importers raise ImportError("Relative module names not supported") ImportError: Relative module names not supported