Giter Site home page Giter Site logo

shipwright's Introduction

Shipwright builds shared Docker images within a git repository in the right order and publishes them tagged with git's revision/branch information so you'll never lose track of an image's origin.

It's the perfect tool for building and publishing your images to places like Docker Hub or your own private registry. Have a look at our motivation to see why we built it and the pain points it solves for you.

Alternatives

See also https://github.com/moby/buildkit

Installation

Shipwright is a simple python script you can install with pip

$ pip install shipwright

Quickstart

Once installed, simply change to a project of yours that contains multiple Dockerfiles and is in git.

Add a json formatted file named .shipwright.json to the root directory of your project. At minimum it should contain the version number of your Shipwright config and a namespace which is either your docker hub user name or the URL to your private repository.

1.0 is the current version for the config.

{
  "version": 1.0,
  "namespace": "[your docker hub name or private repository]"
}

Additionally your config file can map directory names to alternative docker repositories. For example here is a .shipwright.json for the docker hub user shipwright that also maps the root of the git repository to the docker image shipwright/shared and the /foo directory to shipwright/awesome_sauce.

{
  "version": 1.0,

  "namespace": "shipwright",
  "names": {
    "/": "shipwright/shared",
    "/foo": "shipwright/awesome_sauce"
}

Now you can build all the docker images in the git repo by simply changing to any directory under your git repo and running:

$ shipwright

This will recurse through all the directories, looking for ones that contain a Dockerfile. Shipwright will build these Dockerfiles in order and by default tag them with <namespace>/<dirname>:<git commit> along with <namespace>/<dirname>:<git branch> and <namespace>/<dirname>:latest

Working example

We have a sample shipwright project you can use if you want to try this out right away.

$ git clone https://github.com/6si/shipwright-sample.git
$ cd shipwright-sample
$ shipwright 

NOTE: you can use any username you'd like while building locally. In the above example we use ``shipwright``. Nothing is published unless you use the ``push`` command. For your own projects, substitute ``shipwright`` in the above example with your (or your organizations) official docker hub username or private repository.

Notice that if you run the shipwright a second time it will return immediately without doing anything. Shipwright is smart enough to know nothing has changed.

Shipwright really shines when you switch git branches.

$ git checkout new_feature
$ shipwright

Notice that shipwright only rebuilt the shared library and service1, ignoring the other projects because they have a common git ancestry. Running docker images however shows that all the images in the git repository have been tagged with the latest git revision, branch and latest.

In fact, as Shipwright builds images it rewrites the Dockerfiles so that they require the base images with tags from the current git revision. This ensures that the entire build is deterministic and reproducible.

Building

By default, if you run shipwright with no arguments, it will build all Dockerfiles in your git repo. You can specify one or more specifiers to select fewer images to build. For example you can build a single images and its dependencies by simply specifying its name on the command line.

$ shipwright <namespace>/some_image

Run `shipwright --help' for more examples of specifiers and their uses.

Publishing

With one command Shipwright can build your images and push them to a remote repository.

$ shipwright push

If you like you can just push your latest images without building.

$ shipwright push --no-build 

The same specifiers for building also work with push. You might use this to build an entire tree in one step then push a specific image like so.

$ shipwright build
$ shipwright push -e <namespace>/public_image

shipwright's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shipwright's Issues

Random-ish 'raise JSONDecodeError("Extra data", s, end) error during building'

While trying to get shipwright working for the first time, I am finding that builds consistently fail at a semi random spot it the build with an error similar to this:

Traceback (most recent call last):
  File "/Users/skane/virtualenvs/default-3.5/bin/shipwright", line 11, in <module>
    sys.exit(main())
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/cli.py", line 183, in main
    new_style_args=arguments,
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/cli.py", line 270, in run
    for event in events:
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/base.py", line 25, in _build
    for evt in build.do_build(client, ref, targets, pull_cache):
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/build.py", line 42, in do_build
    for evt in build(client, parent_ref, target, pull_cache):
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/build.py", line 96, in build
    yield process_event_(evt)
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/build.py", line 60, in process_event_
    evt_parsed = json_loads(evt)
  File "/Users/skane/virtualenvs/default-3.5/lib/python3.5/site-packages/shipwright/compat.py", line 15, in json_loads
    return json.loads(s, **kwargs)
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/lib/python3.5/json/decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 106)

Sometimes this is almost immediately and sometimes this is a few steps into the Dockerfile.

Our registry is running a current release for Docker Distribution, and I am using the Docker for Mac client at the moment (Docker 1.12).

# cat .shipwright.json
{
  "version": 1.0,
  "namespace": "cf-registry.example.net"
}

# python --version
Python 3.5.2

# pip list
docker-py (1.9.0)
gitdb (0.6.4)
GitPython (2.0.8)
pip (8.1.2)
requests (2.11.1)
setuptools (18.0.1)
shipwright (0.6.3)
six (1.10.0)
smmap (0.9.0)
websocket-client (0.37.0)
wheel (0.24.0)

Unable to install

Tried on both Mac Os X and cent 6, yet shipwright is not in pip. Is this project dead?

Mac

pip install shipwright
Downloading/unpacking shipwright
  Could not find any downloads that satisfy the requirement shipwright
Cleaning up...
No distributions at all found for shipwright
Storing debug log for failure in /Users/sowen/Library/Logs/pip.log

Cent

pip install shipwright
Downloading/unpacking shipwright
  Could not find any downloads that satisfy the requirement shipwright
No distributions at all found for shipwright
Storing complete log in /root/.pip/pip.log

No such file or directory

Ran the sample commands:

$ git clone https://github.com/6si/shipwright-sample.git
$ cd shipwright-sample
$ shipwright shipwright

and got this result...

~/htdocs/shipwright-sample[master]: shipwright shipwright
Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.0.2', 'console_scripts', 'shipwright')()
  File "/usr/local/lib/python2.7/site-packages/shipwright/cli.py", line 63, in main
    for t, docker_commit in Shipwright(namespace,repo,client).build(highlight):
  File "/usr/local/lib/python2.7/site-packages/shipwright/__init__.py", line 64, in build
    current, targets = dependencies.needs_building(self.targets())
  File "/usr/local/lib/python2.7/site-packages/shipwright/__init__.py", line 33, in targets
    docker.tags_from_containers(self.docker_client, containers)  # list of tags
  File "/usr/local/lib/python2.7/site-packages/shipwright/fn.py", line 40, in curry_
    return f(*chain(args, a), **k)
  File "/usr/local/lib/python2.7/site-packages/shipwright/docker.py", line 36, in tags_from_containers
    containers
  File "/usr/local/lib/python2.7/site-packages/shipwright/fn.py", line 56, in compose_
    return reduce(lambda v, f: f(v), ordered, v)
  File "/usr/local/lib/python2.7/site-packages/shipwright/fn.py", line 56, in <lambda>
    return reduce(lambda v, f: f(v), ordered, v)
  File "/usr/local/lib/python2.7/site-packages/shipwright/fn.py", line 40, in curry_
    return f(*chain(args, a), **k)
  File "/usr/local/lib/python2.7/site-packages/shipwright/docker.py", line 29, in last_built_from_docker
    )(client.images(name))
  File "/usr/local/lib/python2.7/site-packages/docker/client.py", line 562, in images
    res = self._result(self._get(self._url("/images/json"), params=params),
  File "/usr/local/lib/python2.7/site-packages/docker/client.py", line 75, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 469, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/requests/adapters.py", line 407, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', error(2, 'No such file or directory'))

Using Python 2.7.8 and shipwright 0.0.2

README.md shows malformed JSON in example

In the README.md, the example given for a .shipwright.json that includes maps is not well-formed JSON:

{
  "version": 1.0,

  "namespace": "shipwright",
  "names": {
    "/": "shipwright/shared",
    "/foo": "shipwright/awesome_sauce"
}

This should be:

{
  "version": 1.0,

  "namespace": "shipwright",
  "names": {
    "/": "shipwright/shared",
    "/foo": "shipwright/awesome_sauce"
  }
}

Create a docker version of shipwright

To speed up setup, dockerize shipwright so that new users can:

$ docker run -v /var/run/docker.sock:/tmp/docker.sock -v <code repo>:/build_dir sixsense/shipwright

Flag --x-assert-hostname requirement for OSX users in the README

It took me quite a while to work out what was going wrong when I tried to run this on OSX. The solution was the --x-assert-hostname flag, which is of course mentioned in the --help documentation. However, I was unaware that the help was there (oops), and I had been following the documentation from the main website. Might be a good idea to flag that at the top? Save some other people the hassle that I had.

docker pull all images for current branch and master before building

Because our buildserver forgets the docker cache between builds we pull the previous build for all the images.

it would be great if we could get shipwright to do it.

Otherwise a command like "shipright images" which lists all the images that shipwright would build would let us write our own command to do this.

Whenever I try to run shipwright it throws the following error

shipwright shipwright
Traceback (most recent call last):
File "/usr/bin/shipwright", line 5, in
from pkg_resources import load_entry_point
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 2655, in
working_set.require(requires)
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: requests>=2.2.1

Using --pull-cache causes a NotFound exception for missing tags if DOCKER_HOST is set.

Hello,

We use Shipwright within a docker-in-docker configuration to build/push our images. With that setup, DOCKER_HOST is set and points to the tcp address of the dind container. It seems the Docker Python library behaves differently in that case versus when using /var/run/docker.sock locally. When doing a docker.APIClient.pull() and the tag isn't present, it returns a docker.errors.NotFound exception instead of a JSON object, and causes Shipwright to die. See for example this traceback:

 /singularity-dockerfiles # shipwright build --pull-cache
/usr/lib/python2.7/site-packages/docker/api/client.py:147: UserWarning: The minimum API version supported is 1.21, but you are using version 1.18. It is recommended you either upgrade Docker Engine or use an older version of Docker SDK for Python.
  'Python.'.format(MINIMUM_DOCKER_API_VERSION, self._version)
Traceback (most recent call last):
  File "/usr/bin/shipwright", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/shipwright/cli.py", line 223, in main
    new_style_args=arguments,
  File "/usr/lib/python2.7/site-packages/shipwright/cli.py", line 331, in run
    for event in events:
  File "/usr/lib/python2.7/site-packages/shipwright/base.py", line 24, in _build
    for evt in build.do_build(client, ref, targets, self._cache):
  File "/usr/lib/python2.7/site-packages/shipwright/build.py", line 43, in do_build
    for evt in build(client, parent_ref, target, cache):
  File "/usr/lib/python2.7/site-packages/shipwright/build.py", line 68, in build
    for evt in cache.pull_cache(image):
  File "/usr/lib/python2.7/site-packages/shipwright/cache.py", line 54, in pull_cache
    stream=True,
  File "/usr/lib/python2.7/site-packages/docker/api/image.py", line 345, in pull
    self._raise_for_status(response)
  File "/usr/lib/python2.7/site-packages/docker/api/client.py", line 208, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/lib/python2.7/site-packages/docker/errors.py", line 30, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found for url: http://unlhcc__docker:2375/v1.18/images/create?tag=fc2c5a5a0b7b&fromImage=unlhcc%2Fcuda-ubuntu ("manifest for unlhcc/cuda-ubuntu:fc2c5a5a0b7b not found")

I've written a fix that catches the NotFound exception and raises a CacheMissException to behave the same as when JSON is returned indicating the tag is missing. I'd be happy to open a PR if that would be useful.

Shipwright often fails when build starts from an empty docker local images repository

Hello

We are experiencing some weird behaviors with shipwright, when building from a host which does not have any docker images already available on the local repo.

ecpff/ecp_bamboo_haproxy | Step 0 : FROM ecpff/ecp_base:85c29768586c
ecpff/ecp_bamboo_haproxy | [STATUS] : Pulling repository ecpff/ecp_base
ecpff/ecp_bamboo_haproxy | [ERROR] Error: image ecpff/ecp_base:85c29768586c not found

ecpff/ecp_base | Step 0 : FROM ecpff/ecp_scratch:85c29768586c
ecpff/ecp_base |  ---> 1137614cd2b3
ecpff/ecp_base | Step 1 : "blabla"
ecpff/ecp_base |  ---> Running in ea17fcd2c6d8
ecpff/ecp_base |  ---> d17262206bbb
ecpff/ecp_base | Removing intermediate container fc3c1aecb76e
ecpff/ecp_base | Successfully built 75b1a04a07f6
Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.2.2', 'console_scripts', 'shipwright')()
  File "/usr/local/lib/python2.7/dist-packages/shipwright/cli.py", line 193, in main
    for event in command(*args):
  File "/usr/local/lib/python2.7/dist-packages/shipwright/__init__.py", line 115, in build_tree
    branch
  File "/usr/local/lib/python2.7/dist-packages/shipwright/docker.py", line 54, in tag_containers
    force=True
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 978, in tag
    self._raise_for_status(res)
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 94, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 404 Client Error: Not Found ("no such id: ecpff/ecp_bamboo_haproxy:85c29768586c")

If no image is already available (build from scratch), then:

  • it loops on images, unable to get the parent image [normal]
  • when it gets to the root image, it builds it [seems also normal]
  • then it often fails with a "no such id" error...

If a restart shipwright (several times), it goes further and further until the end of the build process.

I don't have any debug information to provide right now, but if needed just ask and i can generate them.

KeyError: 'tls' with 0.2.1

Using the latest version of shipwright (0.2.1), when I run shipwright on a project (with a .shipwright.json file), I get an error about tls.

This didn't happen with the previous version I was using (which was pip installed from a pre-release commit using sudo pip install git+git://github.com/6si/shipwright.git@b076b5116e77d84957c3d0dd318b48c7c9d42cf7).

~$ shipwright
Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.2.1', 'console_scripts', 'shipwright')()
  File "/usr/local/lib/python2.7/dist-packages/shipwright/cli.py", line 157, in main
    client_cfg['tls'].assert_hostname = assert_hostname
KeyError: 'tls'

From what I've gathered by Googling, this has something to do with docker.py and the usage of some Docker environment variables. However, if I'm running docker locally on a Unix socket, I shouldn't need to worry about DOCKER_CERT_PATH, DOCKER_HOST, and DOCKER_TLS_VERIFY, right?

Can Shipwright relax this TLS information requirement if I'm running docker on a socket?

shipwright won't build dependant images if the image:latest doesn't already exist on docker hub

Hello first of all great tool! I'm happy to replace my home-grown bash script.

I have a number of images in a single git repository, shipwright builds though all of them right until it reaches one that doesn't have a hub.docker repository, at which point it refuses to continue even though it has already built this dependency (In the example below it's docker-compose image).

Even if you create a repository on hub.docker, but don't push an image, it will refuse to build dependant images, creating a bit of a bootstrapping problem, if you want to use shipwright from the very beginning.

Tagging nigelgbanks/base:eab68eb97bf5 to nigelgbanks/base:eab68eb97bf5
Tagging nigelgbanks/python:eab68eb97bf5 to nigelgbanks/python:eab68eb97bf5
Tagging nigelgbanks/ruby:eab68eb97bf5 to nigelgbanks/ruby:eab68eb97bf5
Tagging nigelgbanks/awscli:eab68eb97bf5 to nigelgbanks/awscli:eab68eb97bf5
Tagging nigelgbanks/docker-compose:eab68eb97bf5 to nigelgbanks/docker-compose:eab68eb97bf5
Tagging nigelgbanks/shipwright:eab68eb97bf5 to nigelgbanks/shipwright:eab68eb97bf5
Tagging nigelgbanks/tugboat:eab68eb97bf5 to nigelgbanks/tugboat:eab68eb97bf5
nigelgbanks/ansible | Step 1 : FROM nigelgbanks/docker-compose
nigelgbanks/ansible | [STATUS] : Pulling repository docker.io/nigelgbanks/docker-compose
nigelgbanks/ansible | [ERROR] Error: Status 503 trying to pull repository nigelgbanks/docker-compose: "<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n\n"

Shipwright 0.0.2 Don't work with private repository

To use private repository we need to use url for DOCKER_HUB_ACCOUNT like foo-foo.com

If we have the following folder

bar/Dockerfile --> From debian:stable
baz/Dockerfile ---> From foo-foo.com/bar

Command : shipwright foo-foo.com
Expected result:
foo-foo.com/bar:master
foo-foo.com/baz:master

Actual result:
...
foo-foo.com/bar | Succesfully built 5cde...
foo-foo.com/baz | [STATUS] : Pulling repository foo-foo.com/bar
foo-foo.com/baz | [ERROR] Error: image bar not found

Shipwright try to pull from private registry instead of using local cache.

Fails on docker-mac due to incorrect docker version reported

Trying to run shipwright on Docker for Mac:

/usr/local/lib/python2.7/site-packages/docker/api/client.py:151: UserWarning: The minimum API version supported is 1.21, but you are using version 1.18. It is recommended you either upgrade Docker Engine or use an older version of Docker SDK for Python. 'Python.'.format(MINIMUM_DOCKER_API_VERSION, self._version) Aborting build, due to uncommitted changes. If you are not ready to commit these changes, re-run with the --dirty flag.

But running docker version shows:
Client:
Version: 1.13.0
API version: 1.25
Go version: go1.7.3
Git commit: 49bf474
Built: Wed Jan 18 16:20:26 2017
OS/Arch: darwin/amd64

Server:
Version: 1.13.0
API version: 1.25 (minimum version 1.12)
Go version: go1.7.3
Git commit: 49bf474
Built: Wed Jan 18 16:20:26 2017
OS/Arch: linux/amd64
Experimental: true`


The server API version is at 1.25 which should be supported

compose-wright command

a tool that has the same command line args as docker-compose:

  • reads a docker-compose config
  • parses out all the "image:" declarations
  • checks if they are in the shipwright config
  • builds only those required images (+deps, and only if there's a new commit or unchecked in changes)
  • runs docker compose, but with the image declartions replaced with the commit specific ones

Error during build, conflict when tagging

Hello

since i upgrade docker to v1.4.1, shipwright has hard time building all my images WHEN there is already built images on the local docker repository.

Example:
after a failed build, i have the following images:

REPOSITORY                      TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
test/image2                  1da5115d0349        fcdf35f64a13        15 hours ago        390.2 MB
test/image1                  1da5115d0349        cb66ae0c4a38        16 hours ago        560.1 MB
test/image0                  1da5115d0349        a0ca4de9ff4c        16 hours ago        109.7 MB
debian                       stable              8633fa92ce62        5 weeks ago         85.1 MB

When i restart the build, i get the following error:

$ shipwright test
Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.0.2', 'console_scripts', 'shipwright')()
  File "/usr/local/lib/python2.7/dist-packages/shipwright/cli.py", line 63, in main
    for t, docker_commit in Shipwright(namespace,repo,client).build(highlight):
  File "/usr/local/lib/python2.7/dist-packages/shipwright/__init__.py", line 75, in build
    this_ref_str
  File "/usr/local/lib/python2.7/dist-packages/shipwright/docker.py", line 47, in tag_containers
    tag=tag
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 1003, in tag
    self._raise_for_status(res)
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 89, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 409 Client Error: Conflict ("Conflict: Tag 1da5115d0349 is already set to image a0ca4de9ff4c2bbfd111ada42168263fc515276e93adf251c1621668bd98de2b, if you want to replace it, please use -f option")

A solution that seems to solve this issue if using the force flag of the docker client tag function
https://github.com/6si/shipwright/blob/master/shipwright/docker.py#L47

Before

client.tag(
container.name + ":" + container.last_built_ref,
container.name,
tag=tag
)

After

client.tag(
container.name + ":" + container.last_built_ref,
container.name,
tag=tag,
force=True
)

Cannot build with boot2docker on OSX

When running shipwright build on OSX with boot2docker I get the following stacktrace:

Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.2.1', 'console_scripts', 'shipwright')()
  File "/Library/Python/2.7/site-packages/shipwright/cli.py", line 190, in main
    for event in command(*args):
  File "/Library/Python/2.7/site-packages/shipwright/__init__.py", line 76, in build
    tree = dependencies.eval(specifiers, self.targets())
  File "/Library/Python/2.7/site-packages/shipwright/__init__.py", line 48, in targets
    docker.tags_from_containers(self.docker_client, containers)  # list of tags
  File "/Library/Python/2.7/site-packages/shipwright/fn.py", line 40, in curry_
    return f(*chain(args, a), **k)
  File "/Library/Python/2.7/site-packages/shipwright/docker.py", line 36, in tags_from_containers
    containers
  File "/Library/Python/2.7/site-packages/shipwright/fn.py", line 56, in compose_
    return reduce(lambda v, f: f(v), ordered, v)
  File "/Library/Python/2.7/site-packages/shipwright/fn.py", line 56, in <lambda>
    return reduce(lambda v, f: f(v), ordered, v)
  File "/Library/Python/2.7/site-packages/shipwright/fn.py", line 40, in curry_
    return f(*chain(args, a), **k)
  File "/Library/Python/2.7/site-packages/shipwright/docker.py", line 29, in last_built_from_docker
    )(client.images(name))
  File "/Library/Python/2.7/site-packages/docker/client.py", line 638, in images
    res = self._result(self._get(self._url("/images/json"), params=params),
  File "/Library/Python/2.7/site-packages/docker/client.py", line 81, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 469, in get
    return self.request('GET', url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 457, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests/sessions.py", line 569, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests/adapters.py", line 420, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: hostname '192.168.59.103' doesn't match 'boot2docker'

boot2docker ip gives 192.168.59.103

Boot2Docker-cli version: v1.4.1 
Git commit: 43241cb
Client version: 1.4.1
Client API version: 1.16
Go version (client): go1.3.3
Git commit (client): 5bc2ff8
OS/Arch (client): darwin/amd64
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

Building without having to commit to git (for dev)

It has taken me quite some time to get things working - this was on account of it not being obvious that you have to commit your changes to git before shipwright will build anything. This behaviour makes the dev cycle a bit annoying if you are rapidly iterating on a container and building it a lot. Any easy way to get shipwright to build, say, a "dev" build from the files in the directory without having to commit first?

Automatically build -xyz dockerfiles

In my multi-docker projects I generally have a layout like:

docker-projects/
├── backend
│   ├── Dockerfile
│   ├── Dockerfile-dev
│   └── Dockerfile-test
└── frontend
    ├── Dockerfile
    ├── Dockerfile-dev
    └── Dockerfile-test

Where the "-dev" builds depend on the vanilla Dockerfile.

It would be great to be able to configure shipwright to automatically pick up these -dev and -test builds as backend, backend-dev, backend-test, frontend, frontend-dev, frontend-test respectively.

maybe with

"dockerfilePattern": "-\\w+"

Feature Request: Force Build (for development workflows)

Currently shipwright will build all the images in a given repository for the current commit. Which is great if you do not plan on making changes within the repository, as subsequent calls to shipwright will not trigger a rebuild if the files are changed.

Often though I have several Dockerfiles in a single repository and I'm actively working on a number of them, making changes and re-building (but not committing).

It would be ideal for me, if I didn't have to think about the build order and I could just have shipwright rebuild the changed images as need be.

Perhaps this could be achieved by adding an additional flag, say -b to always trigger a build. Additionally we could also expose ``--no-cache` from the Docker client as well, it we want Docker to build without the cache.

Pass extra arguments to push/build

it would be great to have something like:

shipwright build -- --pull which runs docker build {container} --pull

and in the JSON:

{
  "version": "1.0.1",

  "namespace": "shipwright",
  "names": {
    "/": "shipwright/shared",
    "/foo": "shipwright/awesome_sauce"
  },
  "buildArgs": {
     "pull": true,
     "ulimit": ["ulimit", "extra ulimit"],
     "cpusetCpus": 2,
  }
}

which runs:

docker build {container} --pull --ulimit=ulimit --ulimit="extra ulimit" --cpuset-cpus 2

shipwright build -t tag removes latest tagging

Steps to reproduce:

  1. shipwright build -t testtag

Expected result

All images are tagged with the same tags as without -t but with the additional testtag

Actual result:

The latest tag is missing

Feature Request: Designate build containers for images

To avoid building images containing build tools, it would be nice to designate that an image is built using another image.

For instance I might create an image that has build-essentials in it from debian/ubuntu and call it sixsense/builder_bob. This image doesn't have to be built by shipwright but it could be.

Then I'd modify .shipwright.json to designate that an image is built inside builder bob like this.

{
  "version": 2.0,

  "namespace": "sixsense",
  "names": {
    "/": {
        "name": "sixsense/buildme",
        "build_with": "sixsense/builder_bob"
     }
   },
  ...
}

Another option is to introduce our own Keyword to the Dockerfile

Examples of this can be seen here.
http://neophob.com/2k014/08/inception-a-docker-image-that-build-docker-images/

Intermittent failures with pushes on 1.10.3

Using 1.10.3 with the Ansible docker_image module. Seeing roughly 4/5 pushes fail with errors like the following:

Error pushing image 127.0.0.1:5000/test-image-3: Extra data: line 2 column 1 - line 3 column 1 (char 64 - 128)

On the other 1/5 it works like a charm.

Seems like there is still a race condition as mentioned in docker/docker-py#1059.

Shipwright crashes

I finally got shipwright installed.

When I try and run any commands, shipwright crashes

spuder@ spuders-MacBook-Pro:/tmp/shipwright$ shipwright spuder
Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 8, in <module>
    load_entry_point('shipwright==0.0.1', 'console_scripts', 'shipwright')()
  File "build/bdist.macosx-10.9-intel/egg/shipwright/cli.py", line 60, in main
  File "build/bdist.macosx-10.9-intel/egg/shipwright/fn.py", line 55, in compose_
  File "build/bdist.macosx-10.9-intel/egg/shipwright/fn.py", line 55, in <lambda>
  File "build/bdist.macosx-10.9-intel/egg/shipwright/fn.py", line 107, in _
  File "build/bdist.macosx-10.9-intel/egg/shipwright/fn.py", line 39, in curry_
  File "build/bdist.macosx-10.9-intel/egg/shipwright/cli.py", line 132, in build
  File "build/bdist.macosx-10.9-intel/egg/shipwright/tar.py", line 130, in mkcontext
  File "build/bdist.macosx-10.9-intel/egg/shipwright/tar.py", line 74, in bundle_docker_dir
TypeError: tar() takes exactly 1 argument (2 given)

spuder@spuders-MacBook-Pro:/tmp/shipwright$ shipwright
Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 8, in <module>
    load_entry_point('shipwright==0.0.1', 'console_scripts', 'shipwright')()
  File "build/bdist.macosx-10.9-intel/egg/shipwright/cli.py", line 24, in main
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py", line 23, in __getitem__
    raise KeyError(key)
KeyError: 'SW_NAMESPACE'

BUG Shipwright 0.2.1 KeyError : 'tls'

Lastest shipwright version install from pip bug with the following error:

 File "/usr/local/bin/shipwright", line 9, in <module>
load_entry_point('shipwright==0.2.1', 'console_scripts', 'shipwright')() 
File "/usr/local/lib/python2.7/dist-packages/shipwright/cli.py", line 157, in main 
client_cfg['tls'].assert_hostname = assert_hostname  
KeyError: 'tls' 

folder naming issues

I just went through some debugging hell with shipwright running on a Jenkins slave. I was trying to get it to build images from a project, but the Jenkins "project name" was not the same as git project name (kind of a long story).

So, the folder ended up being "Test" (with the capital T). Shipwright failed on this in a fairly unfriendly way:

Traceback (most recent call last):
  File "/usr/local/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.2.2', 'console_scripts', 'shipwright')()
  File "/usr/local/lib/python2.7/dist-packages/shipwright/cli.py", line 193, in main
    for event in command(*args):
  File "/usr/local/lib/python2.7/dist-packages/shipwright/__init__.py", line 101, in build_tree
    targets # what needs building
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 295, in _stream_helper
    assert reader._fp.chunked
AssertionError

Which led me on a wild goose chase. I eventually found the issue and switched the Jenkins Git settings to clone to a sub directory by the name of the git project.

It seems that uppercase folder names cause errors. I understand that the "names" property in .shipwright.json can be used to fix this by mapping paths to Docker repository names, but the naming restriction should be documented.

I guess that the git project name could be culled from the remotes, but there's no guarantee that there will be any remotes. Documentation would nice, though :)

Shipwright explodes if Docker build outputs unicode characters.

When trying to run shipwright I am getting the following error:

Traceback (most recent call last):
  File "/home/********/envs/slave/bin/shipwright", line 9, in <module>
    load_entry_point('shipwright==0.2.3', 'console_scripts', 'shipwright')()
  File "/home/********/envs/slave/local/lib/python2.7/site-packages/shipwright/cli.py", line 195, in main
    show_fn(writer(event))
  File "/home/********/envs/slave/local/lib/python2.7/site-packages/shipwright/cli.py", line 228, in highlight_
    print(color_fn(name) + " | " + msg)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 39: ordinal not in range(128)

This specific line that was causing it was a ln -sv command which outputs unicode characters (not quite sure why ln needs to output unicode, but that is besides the point).

I worked around this by just stripping out non-ascii characters, but I'm not sure if that's the best solution for this problem.
this is cli.py line 228 on the latest release of shipwright - 0.2.3

  def highlight_(msg):
    msg = ''.join([i if ord(i) < 128 else ' ' for i in msg])
    print(color_fn(name) + " | " + msg)

Drop support for Python 2.

To all users of shipwright, and @ all contributors/and people with forks:

Supporting both Python 2 and Python 3 is becoming a maintenance burden, and blocking use of the shiny new Python 3 features.

  • nice mypy type annotations
  • async/await
  • yield from
  • f'strings'

I'd like to move shipwright Python 3.x only and would like to gather your input.

  • Do you use Shipwright under Python 2.x?
  • Would you be unable to move to Python 3.x
  • What's the latest Python you'd be able to support? 3.4, 3.5, 3.6?
  • What sort of time-line would be appropriate:
    • 🎉 ASAP
    • 😄 March 2017 (next month)
    • 👍 Dec 2017 (match Django)
    • 👎 Jan 2020 (match CPython 2.7)
    • 😕 Never (match PyPy)

@cwoodcock
@danpalmer
@dayer4b
@dilgerma
@dropcam
@graingert
@jamespic
@jbwinters
@kaeawc
@likhvar
@nigelgbanks
@partkyle
@pierre-moire
@pombredanne
@srobertson
@tescalada
@timtylin

Issue with 'shipwright build' and Docker version 1.12.0, build 8eab29e

▶ shipwright build
Traceback (most recent call last):
File "/Users/user/anaconda/bin/shipwright", line 11, in
sys.exit(main())
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/cli.py", line 170, in main
environ=os.environ,
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/cli.py", line 243, in run
for event in events:
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/base.py", line 22, in _build
for evt in build.do_build(self.docker_client, this_ref_str, targets):
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/build.py", line 42, in do_build
for evt in build(client, parent_ref, target):
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/build.py", line 63, in build
built_tags = docker.last_built_from_docker(client, image.name)
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/docker.py", line 28, in last_built_from_docker
return list([x for i in images for x in key_from_image_info(i)])
File "/Users/user/anaconda/lib/python2.7/site-packages/shipwright/docker.py", line 23, in key_from_image_info
return [key_from_image_name(t) for t in image_info_dict['RepoTags']]
TypeError: 'NoneType' object is not iterable

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.