bazelbuild/rules_docker

Rules for building and handling Docker images with Bazel

rules_docker's Issues

docker_push not working

Build rules:

go_binary(
    name = "greeter_http_server",
    srcs = ["greeter_http.go"],
    deps = [":helloworld_proto_go"] + GRPC_GATEWAY_DEPS,
)

docker_build(
    name = "greeter_http_server_docker",
    base = "@joeshaw_busybox_nonroot//image:image.tar",
    cmd = ["./greeter_http_server"],
    files = [":greeter_http_server"],
    ports = ["8080"],
    repository = "registry.gitlab.com/<blah>",
    symlinks = {
        "/usr/bin/greeter_http_server": "greeter_http_server",
    },
)

docker_push(
    name = "greeter_http_server_docker_push",
    image = ":greeter_http_server_docker",
    registry = "registry.gitlab.com",
    repository = "<blah>/samples/helloworld",
    tag = "dev",
)

Error:

$ bazeldev run samples/helloworld:greeter_http_server_docker_push
INFO: Found 1 target...
Target //samples/helloworld:greeter_http_server_docker_push up-to-date:
  bazel-bin/samples/helloworld/greeter_http_server_docker_push
INFO: Elapsed time: 0.544s, Critical Path: 0.29s

INFO: Running command line: bazel-bin/samples/helloworld/greeter_http_server_docker_push
+ ../pusher/file/pusher.par --name=registry.gitlab.com/<blah>/samples/helloworld:dev --tarball=samples/helloworld/greeter_http_server_docker.tar
ERROR:root:Error during upload of: registry.gitlab.com/<blah>/samples/helloworld:dev
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "../pusher/file/pusher.par/__main__.py", line 71, in <module>
  File "../pusher/file/pusher.par/__main__.py", line 66, in main
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 294, in upload
  File "../pusher/file/pusher.par/concurrent/futures/_base.py", line 398, in result
  File "../pusher/file/pusher.par/concurrent/futures/thread.py", line 55, in run
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 264, in _upload_one
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 186, in _put_blob
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 125, in _monolithic_upload
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 333, in Request
containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '202', 'content-length': '0', 'content-security-policy': "object-src 'none'; script-src 'self' 'unsafe-inline' 'unsafe-eval' piwik.gitlab.com https://www.google.com/recaptcha/ https://www.gstatic.com/recaptcha/; style-src 'self' 'unsafe-inline'; img-src * data: blob:; frame-src 'self' https://www.google.com/recaptcha/; frame-ancestors 'none'; connect-src 'self' wss://gitlab.com; report-uri https://sentry-infra.gitlap.com/api/3/csp-report/?sentry_key=a664fdde83424b43a991f25fa7c78987", 'x-content-type-options': 'nosniff', 'docker-upload-uuid': '9e142cbb-38be-4023-a995-3973871bb035', 'server': 'nginx', 'range': '0-0', 'docker-distribution-api-version': 'registry/2.0', 'location': 'https://registry.gitlab.com/v2/<blah>/samples/helloworld/blobs/uploads/9e142cbb-38be-4023-a995-3973871bb035?_state=e7GilJ0V9PBDIMONe0vaApbNxhosgpwZNLyj1rpjph17Ik5hbWUiOiJjaGFpdGFueWE5MTg2L3NyZWxsaWszL3NhbXBsZXMvaGVsbG93b3JsZCIsIlVVSUQiOiI5ZTE0MmNiYi0zOGJlLTQwMjMtYTk5NS0zOTczODcxYmIwMzUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTctMDUtMDFUMjM6MDI6MjEuMTkxMzE3MTE1WiJ9', 'date': 'Mon, 01 May 2017 23:02:21 GMT', 'content-type': 'text/plain; charset=utf-8'}
: None
ERROR: Non-zero return code '1' from command: Process exited with status 1.

Env details:

$ docker -v
Docker version 17.05.0-ce-rc2, build c57fdb2a14cfba584686ddad909e3006284d10aa

$ sudo cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=17.04
DISTRIB_CODENAME=zesty
DISTRIB_DESCRIPTION="Ubuntu 17.04"
NAME="Ubuntu"
VERSION="17.04 (Zesty Zapus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 17.04"
VERSION_ID="17.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=zesty
UBUNTU_CODENAME=zesty

pushall emits a warning

Using pushall to push a bundle always emits a warning:

docker/contrib/push-all.bzl:39:7: Pushing an image based on a tarball can be very expensive.  If the image is the output of a docker_build, consider dropping the '.tar' extension. If the image is checked in, consider using docker_import instead.

Any way around this?

Creating empty, mode 0777 /tmp directory

Looking in the docs it appears that it would be possible to make a file in a directory called "/tmp" but I can't figure out how to make a /tmp dir with mode 0777. Is that possible?

Stamp for docker_push?

In my docker/BUILD file I have something like this to create docker images. Currently this requires bazel clean between builds for these services because I haven't figured out how to get the stamps to update every run (which is besides the point, there must be a way).

package(default_visibility = ["//visibility:public"])

load("@io_bazel_rules_docker//docker:docker.bzl", "docker_bundle", "docker_push")

DOCKER_BUILDS = {
    "assets":       "//assets/assets:docker", # etc...
}

[docker_bundle(
    name = binary,
    images = {"gcr.io/{GCLOUD_PROJECT_ID}/%s:{BRANCH_NAME}-{BUILD_NUMBER}" % binary: docker_image},
    stamp = True
 ) for binary, docker_image in DOCKER_BUILDS.items()]

My desire is to define matching docker_push targets:

[docker_push(
    name = "push_" + binary,
    image = ":" + binary,
    registry = "gcr.io",
    repository = "{GCLOUD_PROJECT_ID}/%s" % binary,
    tag = "{BRANCH_NAME}-{BUILD_NUMBER}",
    stamp = true,
 ) for binary in DOCKER_BUILDS.keys()]

However this doesn't appear to be supported. I assume there is a reason why that is escaping me. Or maybe I should just be doing this in an external tool.

Allow Ignoring of pre-loaded packages.

Via the docker_repositories() initialization command it would be great if we had a way to re-use existing dependencies.

Let's say for example I was attempting to build something in the tensorflow/tensorflow tf_workspace, that we have loaded up. It turns out they have already imported six:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/workspace.bzl#L302

native.bind(
    name = "six",
    actual = "@six_archive//:six",
)

Without being able to override in some manner we are unable to use the docker_build rules in the same workspace. pubref/rules_protobuf an interesting way to handle these sorts of dependencies:
https://github.com/pubref/rules_protobuf#overriding-or-excluding-workspace-dependencies

go_proto_repositories(
  excludes = [
    "com_github_golang_glog",
  ]
)

Which then uses some require logic under the hood to load / not load the repository functions.

In interim I suppose it's possible to manually include the dependencies other than the ones we have already defined and go from there. This seems like something that might get solved as we move towards: bazelbuild/bazel#1943

bazel explodes on multiple docker_build rules in same package with different external base images

Simple example given in ixdy@8125a4c.

I have two docker_build rules in the same BUILD file, each one depending on an different external image (pulled in through docker_pull in my WORKSPACE).

When I try to build both, bazel explodes, I guess because they are both trying to create the same artifact, image.tar.id:

$ bazel build foo.tar bar.tar
.
ERROR: file 'image.tar.id' is generated by these conflicting actions:
Label: //:foo, //:bar
RuleClass: docker_build_ rule
Configuration: 32a994dd81b79c0621b68fa6003f0e25
Mnemonic: ExtractID
Action key: c0d8c6c6b75540d6052f398dcbf6fc31, fdc7414328621a989a7e2ce8c931afc8
Progress message: ExtractID image.tar.id
PrimaryInput: File:[/usr/local/google/home/jgrafton/.cache/bazel/_bazel_jgrafton/1c344542258c3021719db9a8e772b13f[source]]external/official_python/image/image.tar, File:[/usr/local/google/home/jgrafton/.cache/baz
el/_bazel_jgrafton/1c344542258c3021719db9a8e772b13f[source]]external/official_busybox/image/image.tar
PrimaryOutput: File:[[/usr/local/google/home/jgrafton/.cache/bazel/_bazel_jgrafton/1c344542258c3021719db9a8e772b13f/execroot/rules_docker]bazel-out/local-fastbuild/bin]image.tar.id
.
ERROR: Analysis of target '//:foo' failed; build aborted.
INFO: Elapsed time: 35.226s

Migrate documentation to skydoc

See #30 for more details.

Make testing/e2e.sh a bazel test...

So we can run it on Bazel.

We have docker test in Bazel, and we can depends on the bazel installer to get bazel inside the test so it should be possible.

docker_build doesn't allow '--' in entrypoint

See original issue: GoogleContainerTools/distroless#56

Unable to pull from index.docker.io

docker_pull(
  name = "java_base",
  registry = "index.docker.io",
  repository = "openjdk",
  tag = "8"
)

containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '401', 'content-length': '150', 'strict-transport-security': 'max-age=31536000', 'docker-distribution-api-version': 'registry/2.0', 'date': 'Sat, 13 May 2017 17:13:17 GMT', 'content-type': 'application/json; charset=utf-8', 'www-authenticate': 'Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:openjdk:pull",error="insufficient_scope"'}
authentication required: [{u'Action': u'pull', u'Type': u'repository', u'Class': u'', u'Name': u'openjdk'}]

Why is authentication required to do the same as docker pull openjdk:8? Is there a workaround?

if --name is not provided to pusher, pusher should inspect repositories in tar

opening issue here since issues aren't enabled in https://github.com/google/containerregistry

docker_build: unintuitive behavior if data_path isn't a prefix of added files' paths

If data_path is not a prefix of a file, that file gets inserted with its entire path form the repository root in tact. This can actually be exploited as a workaround for the bazelbuild/bazel#677 : set data_path to some absolute path that's not a prefix of the files you are adding.

Top-level docker_build targets are broken again

In the switch to rules_docker I broke top-level docker_build targets because the name we use isn't an acceptable name to the containerregistry.client.docker_name module.

Here's the stack trace:

$ bazel build :image
INFO: Found 1 target...
ERROR: /home/mattmoor/java-docs-samples/flexible/helloworld/BUILD:16:1: null failed: Process exited with status 1 [sandboxed].
Traceback (most recent call last):
  File "/home/mattmoor/.cache/bazel/_bazel_mattmoor/d76411ba60a897ec070f41fe6c460c8f/bazel-sandbox/c3d33b99-a06b-4472-a6a8-c051f811fdf6-0/execroot/helloworld/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image.runfiles/helloworld/../io_bazel_rules_docker/docker/create_image.py", line 198, in <module>
    main()
  File "/home/mattmoor/.cache/bazel/_bazel_mattmoor/d76411ba60a897ec070f41fe6c460c8f/bazel-sandbox/c3d33b99-a06b-4472-a6a8-c051f811fdf6-0/execroot/helloworld/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image.runfiles/helloworld/../io_bazel_rules_docker/docker/create_image.py", line 195, in main
    args.base, args.metadata, args.name, args.repository)
  File "/home/mattmoor/.cache/bazel/_bazel_mattmoor/d76411ba60a897ec070f41fe6c460c8f/bazel-sandbox/c3d33b99-a06b-4472-a6a8-c051f811fdf6-0/execroot/helloworld/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image.runfiles/helloworld/../io_bazel_rules_docker/docker/create_image.py", line 128, in create_image
    repository=repository, tag=name))
  File "/home/mattmoor/.cache/bazel/_bazel_mattmoor/d76411ba60a897ec070f41fe6c460c8f/bazel-sandbox/c3d33b99-a06b-4472-a6a8-c051f811fdf6-0/execroot/helloworld/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image.runfiles/containerregistry/client/docker_name_.py", line 148, in __init__
    super(Tag, self).__init__(parts[0])
  File "/home/mattmoor/.cache/bazel/_bazel_mattmoor/d76411ba60a897ec070f41fe6c460c8f/bazel-sandbox/c3d33b99-a06b-4472-a6a8-c051f811fdf6-0/execroot/helloworld/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image.runfiles/containerregistry/client/docker_name_.py", line 114, in __init__
    raise self._validation_exception(name)
containerregistry.client.docker_name_.BadNameException: Docker image name must be fully qualified (e.g.registry/repository:tag) saw: bazel

docker_push incompatible with quay.io (manifest schema version not supported?)

I'm trying to push an image to quay.io with docker_push, and get the following error:

$ bazel run //python2.7:push_python27_cpp 
INFO: Analysed target //python2.7:push_python27_cpp (0 packages loaded).
INFO: Found 1 target...
Target //python2.7:push_python27_cpp up-to-date:
  bazel-bin/python2.7/push_python27_cpp
INFO: Elapsed time: 0.167s, Critical Path: 0.01s
INFO: Build completed successfully, 1 total action

INFO: Running command line: bazel-bin/python2.7/push_python27_cpp
+ ../pusher/file/pusher.par --name=quay.io/postmates/distroless_python27_cpp:dev --config=python2.7/python27_cpp.config --digest=base/with_tmp-layer.tar.gz.sha256 --digest=base/base-layer.tar.gz.sha256 --digest=python2.7/python27-layer.tar.gz.sha256 --digest=python2.7/python27_cpp-layer.tar.gz.sha256 --layer=base/with_tmp-layer.tar.gz --layer=base/base-layer.tar.gz --layer=python2.7/python27-layer.tar.gz --layer=python2.7/python27_cpp-layer.tar.gz
ERROR:root:Error during upload of: quay.io/postmates/distroless_python27_cpp:dev
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "../pusher/file/pusher.par/__main__.py", line 129, in <module>
  File "../pusher/file/pusher.par/__main__.py", line 124, in main
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 300, in upload
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 228, in _put_manifest
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 340, in Request
containerregistry.client.v2_2.docker_http_.V2DiagnosticException: response: {'status': '415', 'content-length': '131', 'server': 'nginx/1.13.3', 'connection': 'keep-alive', 'date': 'Wed, 02 Aug 2017 19:14:01 GMT', 'content-type': 'application/json'}
manifest invalid: {u'message': u'manifest schema version not supported'}

Language-specific runtime rules

I'm opening this issue to track discussions around the creation of language runtime rules (@dlorenc @sebgoa FYI).

Triage quoting

On #98 Doug pointed out that we likely have various quoting problems passing strings as values to utility binaries.

This tracks triaging and fixing them.

Bad mktemp syntax on Mac OS

On Mac OS X, running a docker_build rule with bazel run gives me

mktemp: option requires an argument -- t
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix 

I'm guessing this comes from

rules_docker/docker/incremental_load.sh.tpl

Lines 42 to 43 in 48a438c

	TEMP_FILES="$(mktemp -t)"
	TEMP_IMAGES="$(mktemp -t)"

.

What was the "review feedback" in 3042c96 ? https://linux.die.net/man/1/mktemp suggests that -t is deprecated.

Kubernetes rules

I am opening this issue to track discussions around what shape rules_k8s might take, and to enumerate the kinds of scenarios folks would like to see rules_k8s cover.

Please add simple helloworld example for bazel Docker Rules

Please update the documentation for bazel docker rules with a concrete, easy to follow live example.

I'm familiar with traditional docker-based build and as a (very) new user of bazel, I'd like to see where i can use bazel.

It'd help if we update our docs with a 'concrete' example: eg. suggest a full end to end example that runs alpine with python and then simply echo python -c 'print "hello world"' or some such; (something i can run off the bat)

Unable to pull from local docker registry

docker_pull does not work with docker registry without authentication. The error message is "external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 128, in CheckState, containerregistry.client.v2_2.docker_http_.BadStateException: Unexpected status: 200".

https://github.com/docker/docker.github.io/blob/master/registry/deploying.md

The docker registry spec states "If a 200 OK response is returned, the registry implements the V2(.1) registry API and the client may proceed safely with other V2 operations." However, containerregistry always expects 401 status code.

https://github.com/docker/distribution/blob/master/docs/spec/api.md#api-version-check
https://github.com/google/containerregistry/blob/master/client/v2_2/docker_http_.py#L201

docker_bundle always up-to-date (nothing to build)

Playing around trying to see how this can be used for a general go project. Building the binary via go_binary and image via docker_build works, but docker_bundle always says it's up-to-date, when no binary bundle (tarball I'm expecting is being produced.

Output is:

$ bazel build :docker

INFO: Found 1 target...
Target //:docker up-to-date (nothing to build)
INFO: Elapsed time: 0.837s, Critical Path: 0.64s

Here's my BUILD file:

go_binary(
    name = "dummy-web",
    library = ":go_default_library",
)

docker_build(
    name = "docker-internal",
    base = "@distroless_base//image:image.tar",
    entrypoint = ["/dummy-web"],
    files = [":dummy-web"],
    visibility = ["//visibility:private"],
)

docker_bundle(
    name = "docker",
    images = {
        "dummy/web:latest": ":docker-internal",
    },
    stamp = True,
)

I assume i'm doing something wrong but not sure what (I'm a bazel n00b I'm afraid).

Supporting RUN instructions?

Skimming through the code, it does not seem that docker_build rules support RUN instructions: https://docs.docker.com/engine/reference/builder/#run

Is there a plan to add these?

Specifically I'd like to run pip install -r requirements.txt. For now my workaround is to build an image with my requirements pre-installed but that of course is very brittle

Docker image tarballs have wrong created time when imported to Docker

$ docker images|grep dummy

dummy/web                                                  latest                    df93a33aa16b        292 years ago       21.1MB

I also noticed that all files in the tarball have a created date of 1970-01-01 01:00:

$ tar tvf bazel-bin/docker.tar

-rw-r--r-- 0/0             606 1970-01-01 01:00 df93a33aa16b04e027f600b2f429e80a8b6a8896d035910ec5b90109269d3326.json
-rw-r--r-- 0/0               3 1970-01-01 01:00 9f8383f973c5c7dece737d92e6587a0f37002b20287303512f260d6a506bf4f4/VERSION
-rw-r--r-- 0/0         9482240 1970-01-01 01:00 9f8383f973c5c7dece737d92e6587a0f37002b20287303512f260d6a506bf4f4/layer.tar
-rw-r--r-- 0/0             349 1970-01-01 01:00 9f8383f973c5c7dece737d92e6587a0f37002b20287303512f260d6a506bf4f4/json
-rw-r--r-- 0/0               3 1970-01-01 01:00 fb8b0e2a7f9d34c6bd92e1700b942fe81e3603d021fc442470cdd8227d3dbc7a/VERSION
-rw-r--r-- 0/0        11950080 1970-01-01 01:00 fb8b0e2a7f9d34c6bd92e1700b942fe81e3603d021fc442470cdd8227d3dbc7a/layer.tar
-rw-r--r-- 0/0             241 1970-01-01 01:00 fb8b0e2a7f9d34c6bd92e1700b942fe81e3603d021fc442470cdd8227d3dbc7a/json
-rw-r--r-- 0/0               3 1970-01-01 01:00 07dd64f81c937fe7a80aabfb2b5376e80746bed42e8a89c74bf2c579b553dd50/VERSION
-rw-r--r-- 0/0           10240 1970-01-01 01:00 07dd64f81c937fe7a80aabfb2b5376e80746bed42e8a89c74bf2c579b553dd50/layer.tar
-rw-r--r-- 0/0             178 1970-01-01 01:00 07dd64f81c937fe7a80aabfb2b5376e80746bed42e8a89c74bf2c579b553dd50/json
-rw-r--r-- 0/0              95 1970-01-01 01:00 repositories
-rw-r--r-- 0/0             367 1970-01-01 01:00 manifest.json

support build stamp with docker_push

I should be able to use a variable from my workspace status to tag a docker image when running docker_push.

links in docker_build docs are busted

The links in the descriptions of the args of docker_build that link to specific anchors on https://docs.docker.com/reference/builder/ are broken now and do not send the user to the specific topic.

e.g. https://docs.docker.com/reference/builder/#volumes on volumes and https://docs.docker.com/reference/builder/#expose on ports

Rewrite incremental_load.sh.template in python

The shell script has grown in complexity and is hard to port to new architecture, we should probably rewrite it in python at some point.

Remove requirement on GNU tar

A patch in #87 explicitly made GNU tar a requirement.

The root cause of the issue that prompted the change is tar usage within incremental_load.sh.tpl. tar --create --absolute-names --dereference ... causes errors when the BSD variant is used (default on Macs).

It would be good to remove the prerequisite on GNU tar within the script so other implementations (especially BSD) could be used without users having to do anything.

Implementing #72 will most likely supersede this.

Feature request: support file deletion on docker_build

From @justinsb:

Could we / should we add the ability to delete files from a docker layer as part of docker_build?

The primary use case I am thinking of is that we install a deb package, but then we remove e.g. the man pages or the locale files, so as to keep the image size small.

Original Issue: bazelbuild/bazel#1335

files added from repository rule have impossible paths

If I add the following to docker/testdata/BUILD:

docker_build(
    name = "data_path_image_external",
    files = ["@com_google_guava_guava//jar"],
    mode = "0644",
    data_path = "/",
)

It produces an image with:

$ tar tvf ../../bazel-bin/docker/testdata/data_path_image_external-layer.tar
drwxr-xr-x 0/0               0 1970-01-01 00:00 ./
tar: Removing leading `./..' from member names
drwxr-xr-x 0/0               0 1970-01-01 00:00 ./../
tar: Removing leading `./../' from member names
drwxr-xr-x 0/0               0 1970-01-01 00:00 ./../com_google_guava_guava/
drwxr-xr-x 0/0               0 1970-01-01 00:00 ./../com_google_guava_guava/jar/
-rw-r--r-- 0/0         2256213 1970-01-01 00:00 ./../com_google_guava_guava/jar/guava-18.0.jar

docker_import may result in layer mismatch

Resulting in:

invalid manifest, layers length mismatch: expected 6, got 4

The bug is here:

  TOTAL_DIFF_IDS=($(cat "${name}" | python -mjson.tool | \
      grep sha256 | cut -d'"' -f 2 | cut -d':' -f 2))

Essentially, this doesn't filter things enough. The config may contain lines like:

        "Image": "sha256:3fdb6bcbf1c1e852e13c0defe66743162cd07bee677bdbdd82db7f3f395349e4",

Which results in the manifest.json containing additional entries like "Image.tar".

The fix is to expand the filter to something like: grep -E '^ +"sha256:'

add filter / exclude to docker_build

At the moment, if you install a deb in debian_build, all content from the deb is extracted.
It would be nice if some filter / exclude option was added to e.g filer /usr/share/man , /usr/share/doc

Document the more-efficient on-disk format used by rules_docker

This is a work in progress on top of #70 building tighter integration between docker_pull, docker_push and the new intermediate form of docker_build in the diffbase PR.

For users that want to build and transport Docker images without running a Linux-only daemon as root, the Bazel docker rules and pusher.par are currently the only game in town. Rather than treating this new image layout as an opaque implementation detail of docker_push, consider naming it and documenting its structure.

Then you'd have the additional advantage over docker save and docker push in that people could figure out how to integrate other tooling into the pipeline without having to crack open the Docker CLI source code.

-- from @jmillikin-stripe on #71

docker_build error: AttributeError: 'module' object has no attribute 'ExtractValue'

docker_build rule is failing locally for me. Any idea what could be causing the following error:

INFO: Found 1 target...
ERROR: /home/jdyson/projects/bazel-golang-docker-example/BUILD:20:1: null failed: Process exited with status 1 [sandboxed].
Traceback (most recent call last):
  File "/home/jdyson/.cache/bazel/_bazel_jdyson/0016da4edad2c31f0ea83fd2b70559a0/bazel-sandbox/5912333423796671488/execroot/bazel-golang-docker-example/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image_config.runfiles/bazel_golang_docker_example/../io_bazel_rules_docker/docker/create_image_config.py", line 107, in <module>
    main()
  File "/home/jdyson/.cache/bazel/_bazel_jdyson/0016da4edad2c31f0ea83fd2b70559a0/bazel-sandbox/5912333423796671488/execroot/bazel-golang-docker-example/bazel-out/host/bin/external/io_bazel_rules_docker/docker/create_image_config.runfiles/bazel_golang_docker_example/../io_bazel_rules_docker/docker/create_image_config.py", line 85, in main
    layers.append(utils.ExtractValue(layer))
AttributeError: 'module' object has no attribute 'ExtractValue'
Use --strategy=ImageConfig=standalone to disable sandboxing for the failing actions.
Target //:go_example failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 4.010s, Critical Path: 1.77s

default repository for docker_build

It'd be nice to have a way to change the default repository for my docker_build somewhere in WORKSPACE or a top-level BUILD.bazel or similar. I always forget to change it to my private gcr.io docker repo unless I copy and paste the docker_build from elsewhere.

docker_pull fails on syntax error

Hi,

I'm using ArchLinux and I get an error when adding that to my workspace:

docker_pull(
    name = "busybox",
    registry = "index.docker.io",
    repository = "library/busybox",
    tag = "1.26.2-musl"
)

the error is:

    Pull command failed: Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/__main__.py", line 26, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/containerregistry/client/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 26, in <module>
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 946, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 885, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1157, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1131, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1112, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 427, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 544, in spec_from_file_location
  File "/home/sphax/.cache/bazel/_bazel_sphax/420d00f9ffa546a452fc5bdf768b31b7/external/puller/file/puller.par/httplib2/__init__.py", line 942
    print "connect: (%s, %s) ************" % (self.host, self.port)
                                         ^
SyntaxError: invalid syntax

By default Arch uses Python 3, looks like this is the problem ?

Is there a way to tell bazel to use python 2 ? I tried to run bazel with --python2_path=/usr/bin/python2 when it doesn't work.

Pushing to private registry throws an exception

Hey,

First of all thanks for working on bazel and docker support. I'm new to bazel. I try to figure out whether it makes sense for me to use bazel. At the moment I try building docker images and pushing them to my registry.

I have my own docker registry running, but if I try to push using docker_push an error is thrown:

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "../pusher/file/pusher.par/__main__.py", line 129, in <module>
  File "../pusher/file/pusher.par/__main__.py", line 121, in main
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_session_.py", line 71, in __init__
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 177, in __init__
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 215, in _Ping
  File "../pusher/file/pusher.par/containerregistry/client/v2_2/docker_http_.py", line 128, in _CheckState
containerregistry.client.v2_2.docker_http_.BadStateException: Unexpected "www-authenticate" header: Basic realm="Registry Realm"

My docker command for running the registry is basically:

docker run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  -v `pwd`/auth:/auth \
  -e "REGISTRY_AUTH=htpasswd" \
  -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" \
  -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd \
  -v `pwd`/certs:/certs \
  -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
  -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
  registry:2

My BUILD file looks like this:

load("@io_bazel_rules_docker//docker:docker.bzl", "docker_build", "docker_push")

docker_build(
    name="db1",
    files=["file.txt", "//project/setup:setup"],
    base="@nginx-alpine//image",
)

docker_push(
    name="push-db1",
    image=":db1",
    registry="registry.example.com",
    repository="project/db1",
    tag="latest",
)

My ~/.docker/config.jsonfile:

cat ~/.docker/config.json
{
	"auths": {
		"https://registry.gitlab.com": {},
		"registry.example.com": {},
		"registry.gitlab.com": {}
	},
	"credsStore": "osxkeychain"
}

Running bazel build works fine, but I don't know how to provide the credentials.

Can someone guide me how to provide the credentials?

Thanks!
Tarek

document what happens with docker_build targets when run by bazel build vs run

The old docs at https://bazel.build/versions/master/docs/be/docker.html, I believe, mentioned what would happen when a docker_build target was run with bazel build vs bazel run. I don't see that in the README here, but maybe I missed it?

It'd be nice to have a thing to hand to newbies about these targets.

digest or user input as docker_push tag

So, I'm trying to figure out how to avoid needing docker in my CI in order to make deploys to kubernetes. A docker_push plus some locally kept templated k8s deployment YAMLs or just kubectl set image seems doable, but the default deployment strategies won't work if that tag is always latest (plus the k8s folks recommend against that for pretty good reasons).

My question: Is there a way to specify the docker image's tag in docker_push such that it uses the SHA256 digest of the image?

I guess this is maybe two questions: is there a way to specify what tag = to build with from user input or is there a way to get to the digest from skylark code that specifies the tag attribute?

(As a little side thing here, using the digest as the tag has the nice benefit of making k8s do a rolling restart only occur if there's been a change to the docker image since the last deploy, which is handy!)

Add stamping documentation

We definitely need this for docker_push, but may as well for docker_bundle.

We should have a section specifically dedicated to stamping in the README.md.

expected value of type 'string' for attribute 'base' in 'docker_build_' rule, but got ["@java_base//image:image.tar"] (list).

In the readme, this given example throws an error expected value of type 'string' for attribute 'base' in 'docker_build_' rule, but got ["@java_base//image:image.tar"] (list).

docker_build(
    name = "app",
    # References docker_pull from WORKSPACE (above)
    base = ["@java_base//image:image.tar"],
    files = ["//java/com/example/app:Hello_deploy.jar"],
    cmd = ["Hello_deploy.jar"]
)

Alpine: Can't RUN `apk add ca-certificates`

I'm trying to convert my Dockerfile to bazel. Here is the Dockerfile:

FROM alpine:3.5
RUN apk add --no-cache ca-certificates && update-ca-certificates
COPY my-binary /
ENTRYPOINT ["/my-binary"]

my-binary connects to some HTTPS websites.

Right now I have:

# in WORKSPACE
docker_pull(
   name = "alpine",
   registry = "index.docker.io",
   repository = "library/alpine",
   tag = "3.5",
)

# in BUILD
docker_build(
    name = "my-binary-docker",
    base = "@alpine//image:image.tar",
    files = [":my-binary"],
)

I'm not sure how to replace the RUN part of the file.

link to source code for puller.par and pusher.par in doc somewhere

Would be nice to see the source of these binaries.

bazel run docker_build doesn't work when using stamps

cc @ixdy

repro

$ bazel run //build:kube-controller-manager                                                                                                                                                                                                                                                           
INFO: Found 1 target...
Target //build:kube-controller-manager up-to-date (nothing to build)
INFO: Elapsed time: 7.461s, Critical Path: 5.67s

INFO: Running command line: bazel-bin/build/kube-controller-manager
Loading d10f0ce649a0386cfc6df629ba86375210548a6d4cf62ca52e722213f4d95eda...
Loaded image: bazel/build:busybox
Loading 9bd778a641e0bbd2a59d3a0d11cfa670aeb0b73800db6321491ada6022f86519...
Loaded image: bazel/build:kube-controller-manager-internal
Tagging 9bd778a641e0bbd2a59d3a0d11cfa670aeb0b73800db6321491ada6022f86519 as gcr.io/google_containers/kube-controller-manager:{STABLE_DOCKER_TAG}
Error parsing reference: "gcr.io/google_containers/kube-controller-manager:{STABLE_DOCKER_TAG}" is not a valid repository/tag

docker_repositories() fails if user's workspace already contains "subpar"

We use subpar for packaging Python. When trying to introduce Docker support using WORKSPACE file (following the example in README.md), I get the following:

	File "/home/dmitryb/.cache/bazel/_bazel_dmitryb/8e618220280fa417619d9462a0a41d8d/external/io_bazel_rules_docker/docker/docker.bzl", line 128, in docker_repositories
		native.git_repository(name = "subpar", remote = "https:/...", ...")
Cannot redefine repository after any load statement in the WORKSPACE file (for repository 'subpar').
ERROR: Error evaluating WORKSPACE file.
ERROR: error loading package 'external': Package 'external' contains errors.

Seems like you're loading subpar as a part of docker_repositories(), and it's conflicting with the existing subpar target we already have.

Maybe namespacing it in some way (e.g. rules_docker_subpar) could prevent this?

error getting credentials

Hi, total newbie error here I am sure, but when trying a simple build I get an credentials error:

ERROR: /Users/sebgoa/Desktop/foobar/wordpress/BUILD:3:1: no such package '@bitnami_minideb//image': Pull command failed: Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/__main__.py", line 103, in <module>
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/__main__.py", line 90, in main
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/containerregistry/client/v2_2/docker_image_.py", line 256, in __enter__
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 182, in __init__
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/containerregistry/client/v2_2/docker_http_.py", line 244, in _Refresh
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 65, in Get
  File "/private/var/tmp/_bazel_sebgoa/d4b7150d10cf6c79d4bea4b85c7dc9d0/external/puller/file/puller.par/containerregistry/client/docker_creds_.py", line 155, in suffix
Exception: Error fetching credential for osxkeychain, exit status: 1
credentials not found in native keychain

and here is the BUILD:

load("@io_bazel_rules_docker//docker:docker.bzl", "docker_build")

docker_build(
    name = "barfoo",
    # References docker_pull from WORKSPACE (above)
    base = "@bitnami_minideb//image:image.tar",
    files = [":foobar/foobar.py"],
    cmd = ["foobar.py"]
)

even though I can docker pull.

So how do I set the creds for the BUILD to work ?

thanks

docker_build should have a way to override the default image name

Please take a look to examples (read README.md)
https://github.com/excavador/docker_bazel_naming_problem/tree/according_documentation
https://github.com/excavador/docker_bazel_naming_problem/tree/another_solution

I do not have any way to specify local docker image repository/name

Also very useless to receive tag-name from target-name (I forced to use target name "latest", not very usefulll)

docker_build does not work well with py_binary

Say I have a python rule:

py_binary(
    name = 'main',
    ...
)

And I make a docker build rule that looks like this:

docker_build(
    files = ['//path/to/bin:main'],
    ...
)

If I do a bazel run for this rule and then inspect the contents of the image, i see that the main executable has been copied over but not the directory main.runfiles or the manifest main.runfile_manifest. Without these files the python executable will not work. Am I doing something wrong?

Bazel CI is reporting things broken

FR: Support aliasing the resulting image in docker_build

This is a fork of the discussion in bazelbuild/bazel#616 that is intended to capture the desire for a docker_build kwarg for tagging the resulting image.

Docker disallows use of 64-byte hexadecimal strings for repository name

Minimal repo: https://gist.github.com/jschaf/5fbdc65730fe9cdf7dcf5ddd9c95e0da

bazel run //:hello

The command errors with:

INFO: Found 1 target...
Target //:java up-to-date:
  bazel-bin/java-layer.tar
INFO: Elapsed time: 36.408s, Critical Path: 0.60s

INFO: Running command line: bazel-bin/java
Loading 24c4a0933bbd2f6c65ed283d84502fed4d22c77bacd109353929fea482da6025...
Loaded image ID: sha256:24c4a0933bbd2f6c65ed283d84502fed4d22c77bacd109353929fea482da6025
Tagging 24c4a0933bbd2f6c65ed283d84502fed4d22c77bacd109353929fea482da6025 as bazel/:java
Error parsing reference: "24c4a0933bbd2f6c65ed283d84502fed4d22c77bacd109353929fea482da6025" is not a valid repository/tag: invalid repository name (24c4a0933bbd2f6c65ed283d84502fed4d22c77bacd109353929fea482da6025), cannot specify 64-byte hexadecimal strings

Here's the upstream issue, Docker #20972. The relevant quote is:

In general, we don't want to allow unprefixed 64-character hexadecimal image references. The concern is that a tag like 3240943c9ea3f72db51bea0a2428e83f3c5fa1312e19af017d026f9bcf70f84b could cause ambiguity when there is a content-addressable image ID with a matching digest. The safest way to avoid tags like this from being created or used is to reject references like this at parsing time. Admittedly, rmi behaves a bit differently here. This is not a big concern because rmi can't be used to create an ambiguous tag.

Is there a work around or fix available?