goldmann/docker-squash

Docker image squashing tool

docker-squash

The problem

Docker creates many layers while building the image. Sometimes it's not necessary or desireable to have them in the image. For example a Dockerfile ADD instruction creates a single layer with files you want to make available in the image. The problem arises when these files are only temporary files (for example product distribution that you want to unpack). Docker will carry this unnecessary layer always with the image, even if you delete these files in next layer. This a waste of time (more data to push/load/save) and resources (bigger image).

Squashing helps with organizing images in logical layers. Instead of having an image with multiple (in almost all cases) unnecessary layers - we can control the structure of the image.

Features

• Can squash last n layers from an image
• Can squash from a selected layer to the end (not always possible, depends on the image)
• Support for Docker 1.9 or newer (older releases may run perfectly fine too, try it!)
• Squashed image can be loaded back to the Docker daemon or stored as tar archive somewhere

Installation

From source code

$ pip install --user https://github.com/goldmann/docker-squash/archive/master.zip

From PyPi

$ pip install docker-squash

It is supported on Python 3.6 and above.

Usage

$ docker-squash -h
usage: cli.py [-h] [-v] [--version] [-d] [-f FROM_LAYER] [-t TAG]
              [--tmp-dir TMP_DIR] [--output-path OUTPUT_PATH]
              image

Docker layer squashing tool

positional arguments:
  image                 Image to be squashed

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose output
  --version             Show version and exit
  -f FROM_LAYER, --from-layer FROM_LAYER
                        Number of layers to squash or ID of the layer (or image ID or image name) to squash from.
                        In case the provided value is an integer, specified number of layers will be squashed.
                        Every layer in the image will be squashed if the parameter is not provided.
  -t TAG, --tag TAG     Specify the tag to be used for the new image. If not specified no tag will be applied
  -m MESSAGE, --message MESSAGE
                        Specify a commit message (comment) for the new image.
  -c, --cleanup         Remove source image from Docker after squashing
  --tmp-dir TMP_DIR     Temporary directory to be created and used. This will NOT be deleted afterwards for
                        easier debugging.
  --output-path OUTPUT_PATH
                        Path where the image may be stored after squashing.
  --load-image [LOAD_IMAGE]
                        Whether to load the image into Docker daemon after squashing
                        Default: true

Note that environment variables may be set as documented in here.

License

MIT

Examples

We start with image like this:

$ docker history jboss/wildfly:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
25954e6d2300        3 weeks ago         /bin/sh -c #(nop) CMD ["/opt/jboss/wildfly/bi   0 B
5ae69cb454a5        3 weeks ago         /bin/sh -c #(nop) EXPOSE 8080/tcp               0 B
dc24712f35c4        3 weeks ago         /bin/sh -c #(nop) ENV LAUNCH_JBOSS_IN_BACKGRO   0 B
d929129d4c8e        3 weeks ago         /bin/sh -c cd $HOME     && curl -O https://do   160.8 MB
b8fa3caf7d6d        3 weeks ago         /bin/sh -c #(nop) ENV JBOSS_HOME=/opt/jboss/w   0 B
38b8f85e74bf        3 weeks ago         /bin/sh -c #(nop) ENV WILDFLY_SHA1=c0dd7552c5   0 B
ae79b646b9a9        3 weeks ago         /bin/sh -c #(nop) ENV WILDFLY_VERSION=10.0.0.   0 B
2b4606dc9dc7        3 weeks ago         /bin/sh -c #(nop) ENV JAVA_HOME=/usr/lib/jvm/   0 B
118fa9e33576        3 weeks ago         /bin/sh -c #(nop) USER [jboss]                  0 B
5f7e8f36c3bb        3 weeks ago         /bin/sh -c yum -y install java-1.8.0-openjdk-   197.4 MB
3d4d0228f161        3 weeks ago         /bin/sh -c #(nop) USER [root]                   0 B
f7ab4ea19708        3 weeks ago         /bin/sh -c #(nop) MAINTAINER Marek Goldmann <   0 B
4bb15f3b6977        3 weeks ago         /bin/sh -c #(nop) USER [jboss]                  0 B
5dc1e49f4361        3 weeks ago         /bin/sh -c #(nop) WORKDIR /opt/jboss            0 B
7f0f9eb31174        3 weeks ago         /bin/sh -c groupadd -r jboss -g 1000 && usera   4.349 kB
bd515f044af7        3 weeks ago         /bin/sh -c yum update -y && yum -y install xm   25.18 MB
b78336099045        3 weeks ago         /bin/sh -c #(nop) MAINTAINER Marek Goldmann <   0 B
4816a298548c        3 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
6ee235cf4473        3 weeks ago         /bin/sh -c #(nop) LABEL name=CentOS Base Imag   0 B
474c2ee77fa3        3 weeks ago         /bin/sh -c #(nop) ADD file:72852fc7626d233343   196.6 MB
1544084fad81        6 months ago        /bin/sh -c #(nop) MAINTAINER The CentOS Proje   0 B

And we want to squash all the layers down to layer 4bb15f3b6977.

$ docker-squash -f 4bb15f3b6977 -t jboss/wildfly:squashed jboss/wildfly:latest
2016-04-01 13:11:02,358 root         INFO     docker-scripts version 1.0.0dev, Docker 7206621, API 1.21...
2016-04-01 13:11:02,358 root         INFO     Using v1 image format
2016-04-01 13:11:02,374 root         INFO     Old image has 21 layers
2016-04-01 13:11:02,378 root         INFO     Checking if squashing is necessary...
2016-04-01 13:11:02,378 root         INFO     Attempting to squash last 12 layers...
2016-04-01 13:11:02,378 root         INFO     Saving image 25954e6d230006235eecb7f0cc560264d73146985c2d2e663bac953d660b8730 to /tmp/docker-squash-fbxZz4/old/image.tar file...
2016-04-01 13:11:08,003 root         INFO     Image saved!
2016-04-01 13:11:08,031 root         INFO     Unpacking /tmp/docker-squash-fbxZz4/old/image.tar tar file to /tmp/docker-squash-fbxZz4/old directory
2016-04-01 13:11:08,588 root         INFO     Archive unpacked!
2016-04-01 13:11:08,636 root         INFO     Squashing image 'jboss/wildfly:latest'...
2016-04-01 13:11:08,637 root         INFO     Starting squashing...
2016-04-01 13:11:08,637 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/25954e6d230006235eecb7f0cc560264d73146985c2d2e663bac953d660b8730/layer.tar'...
2016-04-01 13:11:08,637 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/5ae69cb454a5a542f63e148ce40fb9e01de5bb01805b4ded238841bc2ce8e895/layer.tar'...
2016-04-01 13:11:08,637 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/dc24712f35c40e958be8aca2731e7bf8353b9b18baa6a94ad84c6952cbc77004/layer.tar'...
2016-04-01 13:11:08,638 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/d929129d4c8e61ea3661eb42c30d01f4c152418689178afc7dc8185a37814528/layer.tar'...
2016-04-01 13:11:09,113 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/b8fa3caf7d6dc228bf2499a3af86e5073ad0c17304c3900fa341e9d2fe4e5655/layer.tar'...
2016-04-01 13:11:09,115 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/38b8f85e74bfa773a0ad69da2205dc0148945e6f5a7ceb04fa4e8619e1de425b/layer.tar'...
2016-04-01 13:11:09,115 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/ae79b646b9a9a287c5f6a01871cc9d9ee596dafee2db942714ca3dea0c06eef3/layer.tar'...
2016-04-01 13:11:09,115 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/2b4606dc9dc773aa220a65351fe8d54f03534c58fea230960e95915222366074/layer.tar'...
2016-04-01 13:11:09,115 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/118fa9e33576ecc625ebbbfdf2809c1527e716cb4fd5cb40548eb6d3503a75a9/layer.tar'...
2016-04-01 13:11:09,115 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/5f7e8f36c3bb20c9db7470a22f828710b4d28aede64966c425add48a1b14fe23/layer.tar'...
2016-04-01 13:11:10,127 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/3d4d0228f161b67eb46fdb425ad148c31d9944dcb822f67eac3e2ac2effefc73/layer.tar'...
2016-04-01 13:11:10,129 root         INFO     Squashing file '/tmp/docker-squash-fbxZz4/old/f7ab4ea197084ab7483a2ca5409bdcf5473141bfb61b8687b1329943359cc3fe/layer.tar'...
2016-04-01 13:11:10,732 root         INFO     Squashing finished!
2016-04-01 13:11:10,737 root         INFO     New squashed image ID is 52255e75d3eb83123e074f897e8c971dec9d1168a5c82d7c1496a190da2e40ef
2016-04-01 13:11:14,563 root         INFO     Image registered in Docker daemon as jboss/wildfly:squashed
2016-04-01 13:11:14,652 root         INFO     Done

We can now confirm the layer structure:

$ docker history jboss/wildfly:squashed
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
52255e75d3eb        40 seconds ago                                                      358.2 MB
4bb15f3b6977        3 weeks ago         /bin/sh -c #(nop) USER [jboss]                  0 B
5dc1e49f4361        3 weeks ago         /bin/sh -c #(nop) WORKDIR /opt/jboss            0 B
7f0f9eb31174        3 weeks ago         /bin/sh -c groupadd -r jboss -g 1000 && usera   4.349 kB
bd515f044af7        3 weeks ago         /bin/sh -c yum update -y && yum -y install xm   25.18 MB
b78336099045        3 weeks ago         /bin/sh -c #(nop) MAINTAINER Marek Goldmann <   0 B
4816a298548c        3 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
6ee235cf4473        3 weeks ago         /bin/sh -c #(nop) LABEL name=CentOS Base Imag   0 B
474c2ee77fa3        3 weeks ago         /bin/sh -c #(nop) ADD file:72852fc7626d233343   196.6 MB
1544084fad81        6 months ago        /bin/sh -c #(nop) MAINTAINER The CentOS Proje   0 B

Other option is to specify how many layers (counting from the newest layer) we want to squash.Let's squash last 10 layers from the jboss/wildfly:latest image:

$ docker-squash -f 10 -t jboss/wildfly:squashed jboss/wildfly:latest
2016-04-01 13:15:06,488 root         INFO     docker-scripts version 1.0.0dev, Docker 7206621, API 1.21...
2016-04-01 13:15:06,488 root         INFO     Using v1 image format
2016-04-01 13:15:06,504 root         INFO     Old image has 21 layers
2016-04-01 13:15:06,504 root         INFO     Checking if squashing is necessary...
2016-04-01 13:15:06,504 root         INFO     Attempting to squash last 10 layers...
2016-04-01 13:15:06,505 root         INFO     Saving image 25954e6d230006235eecb7f0cc560264d73146985c2d2e663bac953d660b8730 to /tmp/docker-squash-fu80CX/old/image.tar file...
2016-04-01 13:15:12,136 root         INFO     Image saved!
2016-04-01 13:15:12,167 root         INFO     Unpacking /tmp/docker-squash-fu80CX/old/image.tar tar file to /tmp/docker-squash-fu80CX/old directory
2016-04-01 13:15:12,706 root         INFO     Archive unpacked!
2016-04-01 13:15:12,756 root         INFO     Squashing image 'jboss/wildfly:latest'...
2016-04-01 13:15:12,756 root         INFO     Starting squashing...
2016-04-01 13:15:12,756 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/25954e6d230006235eecb7f0cc560264d73146985c2d2e663bac953d660b8730/layer.tar'...
2016-04-01 13:15:12,757 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/5ae69cb454a5a542f63e148ce40fb9e01de5bb01805b4ded238841bc2ce8e895/layer.tar'...
2016-04-01 13:15:12,757 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/dc24712f35c40e958be8aca2731e7bf8353b9b18baa6a94ad84c6952cbc77004/layer.tar'...
2016-04-01 13:15:12,757 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/d929129d4c8e61ea3661eb42c30d01f4c152418689178afc7dc8185a37814528/layer.tar'...
2016-04-01 13:15:13,234 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/b8fa3caf7d6dc228bf2499a3af86e5073ad0c17304c3900fa341e9d2fe4e5655/layer.tar'...
2016-04-01 13:15:13,235 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/38b8f85e74bfa773a0ad69da2205dc0148945e6f5a7ceb04fa4e8619e1de425b/layer.tar'...
2016-04-01 13:15:13,235 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/ae79b646b9a9a287c5f6a01871cc9d9ee596dafee2db942714ca3dea0c06eef3/layer.tar'...
2016-04-01 13:15:13,235 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/2b4606dc9dc773aa220a65351fe8d54f03534c58fea230960e95915222366074/layer.tar'...
2016-04-01 13:15:13,236 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/118fa9e33576ecc625ebbbfdf2809c1527e716cb4fd5cb40548eb6d3503a75a9/layer.tar'...
2016-04-01 13:15:13,236 root         INFO     Squashing file '/tmp/docker-squash-fu80CX/old/5f7e8f36c3bb20c9db7470a22f828710b4d28aede64966c425add48a1b14fe23/layer.tar'...
2016-04-01 13:15:14,848 root         INFO     Squashing finished!
2016-04-01 13:15:14,853 root         INFO     New squashed image ID is fde7edd2e5683c97bedf9c0bf52ad5150db5650e421de3d9293ce5223b256455
2016-04-01 13:15:18,963 root         INFO     Image registered in Docker daemon as jboss/wildfly:squashed
2016-04-01 13:15:19,059 root         INFO     Done

Let's confirm the image structure now:

$ docker history jboss/wildfly:squashed
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
fde7edd2e568        32 seconds ago                                                      358.2 MB
3d4d0228f161        3 weeks ago         /bin/sh -c #(nop) USER [root]                   0 B
f7ab4ea19708        3 weeks ago         /bin/sh -c #(nop) MAINTAINER Marek Goldmann <   0 B
4bb15f3b6977        3 weeks ago         /bin/sh -c #(nop) USER [jboss]                  0 B
5dc1e49f4361        3 weeks ago         /bin/sh -c #(nop) WORKDIR /opt/jboss            0 B
7f0f9eb31174        3 weeks ago         /bin/sh -c groupadd -r jboss -g 1000 && usera   4.349 kB
bd515f044af7        3 weeks ago         /bin/sh -c yum update -y && yum -y install xm   25.18 MB
b78336099045        3 weeks ago         /bin/sh -c #(nop) MAINTAINER Marek Goldmann <   0 B
4816a298548c        3 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
6ee235cf4473        3 weeks ago         /bin/sh -c #(nop) LABEL name=CentOS Base Imag   0 B
474c2ee77fa3        3 weeks ago         /bin/sh -c #(nop) ADD file:72852fc7626d233343   196.6 MB
1544084fad81        6 months ago        /bin/sh -c #(nop) MAINTAINER The CentOS Proje   0 B

docker-squash's Issues

Squashing hangs and times out

2016-02-18 22:46:36,731 root         INFO     Squashing image 'rhscl/python-34-rhel7-candidate'...
2016-02-18 22:46:36,810 root         INFO     Old image has 18 layers
2016-02-18 22:46:36,810 root         INFO     Attempting to squash from layer 2ceccd5b1bd58dfa66cc9326d5e811141b351a116f414e9ed4166bf43337c256...
2016-02-18 22:46:36,810 root         INFO     Checking if squashing is necessary...
2016-02-18 22:46:36,810 root         INFO     We have 11 layers to squash
2016-02-18 22:46:36,811 root         INFO     Saving image 4695cb8451fc1d8de3bdedf5cab2af971a464114fbb34b42a3dab535a27e446b to /tmp/docker-squash-ElGSYo/image.tar file...
2016-02-18 22:56:36,912 root         ERROR    UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=600)
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 68, in _save_image
    image = self.docker.get_image(image_id)
  File "/root/.local/lib/python2.7/site-packages/docker_py-1.2.3-py2.7.egg/docker/utils/decorators.py", line 15, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/root/.local/lib/python2.7/site-packages/docker_py-1.2.3-py2.7.egg/docker/client.py", line 618, in get_image
    stream=True)
  File "/root/.local/lib/python2.7/site-packages/docker_py-1.2.3-py2.7.egg/docker/client.py", line 106, in _get
    return self.get(url, **self._set_request_timeout(kwargs))

I see this on a machine where i'm performing a series of squashes (@mfojtik this is from our push job when it builds all the images... seems like things go ok for a while, and then something goes bad. I've ssh'd into the machine and no squash operations work once it gets in this state, even ones the job did successfully earlier. Restarting the docker daemon does not help, either. Can you work with @goldmann tomorrow to see if you can figure out what's going on? Otherwise we're going to have to disable squashing to get our images rebuilt)

I can also confirm we're not out of disk space on the machine.

Make sure the VERSION file exists in the squashed layer

Currently we do not produce the VERSION file for the squashed layer, but we should to mimic the behavior of Docker.

Gracefully exit when Docker client is not able to connect

We need to exit with a message instead of showing the stacktrace.

squash: populate Size field in metadata

...so docker can display correct size in docker images output

squash: can't squash until base

$ docker history 28206e3d4acd
IMAGE               CREATED             CREATED BY                                      SIZE
28206e3d4acd        13 days ago                                                         0 B
b4747060df20        13 days ago                                                         0 B
df1f40700443        13 days ago                                                         76 B
2ddd275a8578        13 days ago                                                         181 B
2b5560a34cb8        13 days ago                                                         2.305 kB
f594749a6a1e        13 days ago                                                         0 B
54f1460b752c        13 days ago                                                         0 B
e957524d1847        13 days ago                                                         0 B
532aa50000a8        13 days ago                                                         0 B
04e66d3658b1        13 days ago                                                         0 B
34851b543837        13 days ago                                                         0 B
a1f1ce61f39f        13 days ago                                                         0 B
104c16516339        13 days ago                                                         42.68 MB
7452e0cbc31e        13 days ago                                                         14.28 MB
b8b39cb4e218        13 days ago                                                         0 B
10acc31def5d        12 weeks ago                                                        154.1 MB

$ python squash.py -t ./tmp 28206e3d4acd 10acc31def5d new-image
2015-05-06 09:19:33,042 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/28206e3d4acd/json HTTP/1.1" 200 1443
2015-05-06 09:19:33,045 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/10acc31def5d/json HTTP/1.1" 200 1129
2015-05-06 09:19:33,048 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/28206e3d4acd7c86c653121c5be1e7eb69c2273c16f30bb3a372dffa1b4a9e03/json HTTP/1.1" 200 1443
2015-05-06 09:19:33,052 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/b4747060df20ce2aa7ef1341802a6001add0d49756a68e611727e858244820c3/json HTTP/1.1" 200 1511
2015-05-06 09:19:33,056 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/df1f407004432df7c1941a7c36412a145a2c4dca80f4441893db058979f3f774/json HTTP/1.1" 200 1511
2015-05-06 09:19:33,060 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/2ddd275a857816189c3b286634a8782ecd3867336b4d4a00a538e8870034ead7/json HTTP/1.1" 200 1510
2015-05-06 09:19:33,064 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/2b5560a34cb8dc7da06b4b392925aacc0c219e3deabdc81b5390e2a5b399ea2f/json HTTP/1.1" 200 1511
2015-05-06 09:19:33,067 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/f594749a6a1ecfecea8fee63c901fa919585f491cfa782c54bcc0a1bcd53db0f/json HTTP/1.1" 200 1850
2015-05-06 09:19:33,071 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/54f1460b752c02f9c980b7db437ad4613206b8a20089f9f7a4390fbfb78369de/json HTTP/1.1" 200 1537
2015-05-06 09:19:33,075 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/e957524d1847b14f5c1d53ca5414bb8685aec8b4958aa1ca207b885b2b73153e/json HTTP/1.1" 200 1533
2015-05-06 09:19:33,078 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/532aa50000a8e1be43f848f5cc6d5188ab59c54b3944a0f4af78a4e808d774e7/json HTTP/1.1" 200 1431
2015-05-06 09:19:33,081 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/04e66d3658b19086a3920b6d88d77cb793b03750892296b7de4454faff7a8f84/json HTTP/1.1" 200 1433
2015-05-06 09:19:33,084 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/34851b5438378d4b781738eaf2947d6fd554c1129adb196cfa395bd8f8f8c2fd/json HTTP/1.1" 200 1438
2015-05-06 09:19:33,087 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/a1f1ce61f39f0b6f2e866901fa2a71dce023fb804b98e9063b0f8e8e52125e24/json HTTP/1.1" 200 1448
2015-05-06 09:19:33,092 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/104c16516339ad982533c8c3bf64a23af6eada76066847949642cf5e986112eb/json HTTP/1.1" 200 1897
2015-05-06 09:19:33,096 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/7452e0cbc31eeee50bc42f4c057517ba69715b8e76125d97f4998dabcabfcdcd/json HTTP/1.1" 200 1858
2015-05-06 09:19:33,099 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/b8b39cb4e218a20d0dddee3e6977622311c67469e61c2c9395d525bf347b0579/json HTTP/1.1" 200 1436
2015-05-06 09:19:33,101 requests.packages.urllib3.connectionpool DEBUG    "GET /run/docker.sock/v1.17/images/10acc31def5d6f249b548e01e8ffbaccfd61af0240c17315a7ad393d022c5ca2/json HTTP/1.1" 200 1129
2015-05-06 09:19:33,102 root         ERROR    There are no layers to squash, aborting

Traceback with docker-1.10

In docker-1.10 the format created by docker save surprises docker-scripts, causing a traceback.

[root@localhost ~]# docker build -t hello-world --rm .
Sending build context to Docker daemon 849.3 MB
Step 1 : FROM fedora
 ---> 7f514965bc71
Step 2 : MAINTAINER Tim Waugh <[email protected]>
 ---> Running in 252c27f858a8
 ---> f154af43571d
Removing intermediate container 252c27f858a8
Step 3 : ENV env foo
 ---> Running in 8868d683113f
 ---> ca23b29d5e98
Removing intermediate container 8868d683113f
Step 4 : CMD printf hello world
 ---> Running in 5fef2b292ca8
 ---> 04180b30b127
Removing intermediate container 5fef2b292ca8
Successfully built 04180b30b127
[root@localhost ~]# docker history hello-world
IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
04180b30b127        About a minute ago   /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "printf   0 B                 
ca23b29d5e98        About a minute ago   /bin/sh -c #(nop) ENV env=foo                   0 B                 
f154af43571d        About a minute ago   /bin/sh -c #(nop) MAINTAINER Tim Waugh <twaug   0 B                 
7f514965bc71        11 days ago          /bin/sh -c #(nop) ADD file:f867e28f7b12f9d64d   204 MB              
86c04d55e004        4 months ago         /bin/sh -c #(nop) MAINTAINER Adam Miller <max   0 B                 
[root@localhost ~]# docker-scripts squash hello-world -t hello-world-squashed -f f154af43571d
2015-12-02 11:55:50,010 root         INFO     Running version 0.4.4, Docker 669c279-dirty, API 1.22...
2015-12-02 11:55:50,010 root         INFO     Squashing image 'hello-world'...
2015-12-02 11:55:50,023 root         INFO     Old image has 5 layers
2015-12-02 11:55:50,023 root         INFO     Attempting to squash from layer sha256:f154af43571daa042210fab21c7dd7ee2b31623bc5ab47547c82be80bbf4e11d...
2015-12-02 11:55:50,023 root         INFO     Checking if squashing is necessary...
2015-12-02 11:55:50,023 root         INFO     We have 2 layers to squash
2015-12-02 11:55:50,025 root         INFO     Saving image sha256:04180b30b127c416739b3408a926220cf13dbe44833a1b8c7c9114a7c9c2e298 to /tmp/docker-squash-yVCfjY/image.tar file...
2015-12-02 11:55:54,616 root         INFO     Image saved!
2015-12-02 11:55:54,628 root         INFO     Unpacking /tmp/docker-squash-yVCfjY/image.tar tar file to /tmp/docker-squash-yVCfjY/old directory
2015-12-02 11:55:54,782 root         INFO     Archive unpacked!
2015-12-02 11:55:54,782 root         INFO     New layer ID for squashed content will be: a551939d6a79381de09c6fe55c97eba952efaccd24240f237c747c7566b8e30b
2015-12-02 11:55:54,782 root         INFO     Starting squashing...
2015-12-02 11:55:54,782 root         INFO     Squashing layer 'sha256:04180b30b127c416739b3408a926220cf13dbe44833a1b8c7c9114a7c9c2e298'...
Traceback (most recent call last):
  File "/usr/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.4.4', 'console_scripts', 'docker-scripts')()
  File "/usr/lib/python2.7/site-packages/docker_scripts/cli.py", line 96, in run
    cli.run()
  File "/usr/lib/python2.7/site-packages/docker_scripts/cli.py", line 88, in run
    args.func(args)
  File "/usr/lib/python2.7/site-packages/docker_scripts/cli.py", line 31, in run_squash
    from_layer=args.from_layer, tag=args.tag, output_path=args.output_path, tmp_dir=args.tmp_dir).run()
  File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 497, in run
    layers_to_squash, layers_to_move, squashed_tar, old_image_dir)
  File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 337, in _squash_layers
    with tarfile.open(layer_tar_file, 'r', format=tarfile.PAX_FORMAT) as layer_tar:
  File "/usr/lib64/python2.7/tarfile.py", line 1667, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/lib64/python2.7/tarfile.py", line 1732, in gzopen
    fileobj = gzip.GzipFile(name, mode, compresslevel, fileobj)
  File "/usr/lib64/python2.7/gzip.py", line 94, in __init__
    fileobj = self.myfileobj = __builtin__.open(filename, mode or 'rb')
IOError: [Errno 2] No such file or directory: u'/tmp/docker-squash-yVCfjY/old/sha256:04180b30b127c416739b3408a926220cf13dbe44833a1b8c7c9114a7c9c2e298/layer.tar'

RFE: don't import squashed image to docker, instead provide path to tarball

We're planning to push squashed image to pulp registry. Pulp understands tarballs, not docker images. Therefore we could optimize our workflow from:

1. squash image
   • export image as tarball from docker
   • squash it
   • import tarball as image back to docker
2. export image as tarball from docker
3. compress it
4. push to pulp

to

1. squash image
   • export image as tarball from docker
   • squash it
2. compress tarball (squashed image)
3. push to pulp

Therefore, would it be possible to create a switch or something for such functionality?

Retry a few times when exporting the image in case of timeout

Sometimes we can see timeouts when exporting the image. This is because the Docker daemon can be busy, especially when we execute a lot of builds in parallel. We need to gracefully handle it.

`-t` is required for squash command

using git master:

$ PYTHONPATH=$PWD python ./docker_scripts/cli.py squash image
2016-03-29 09:12:22,124 root         INFO     docker-scripts version 1.0.0rc1, Docker c3959b1, API 1.22...
2016-03-29 09:12:22,124 root         INFO     Using v2 image format
Traceback (most recent call last):
  File "./docker_scripts/cli.py", line 109, in <module>
    run()
  File "./docker_scripts/cli.py", line 105, in run
    cli.run()
  File "./docker_scripts/cli.py", line 91, in run
    args.func(args)
  File "./docker_scripts/cli.py", line 32, in run_squash
    from_layer=args.from_layer, tag=args.tag, output_path=args.output_path, tmp_dir=args.tmp_dir, development=args.development).run()
  File "/home/tt/g/docker-scripts/docker_scripts/squash.py", line 59, in run
    return self.squash(image)
  File "/home/tt/g/docker-scripts/docker_scripts/squash.py", line 71, in squash
    new_image_id = image.squash()
  File "/home/tt/g/docker-scripts/docker_scripts/image.py", line 66, in squash
    self._before_squashing()
  File "/home/tt/g/docker-scripts/docker_scripts/v2_image.py", line 15, in _before_squashing
    super(V2Image, self)._before_squashing()
  File "/home/tt/g/docker-scripts/docker_scripts/image.py", line 143, in _before_squashing
    self.image_name, self.image_tag = self._parse_image_name(self.tag)
  File "/home/tt/g/docker-scripts/docker_scripts/image.py", line 361, in _parse_image_name
    if ':' in image and '/' not in image.split(':')[-1]:
TypeError: argument of type 'NoneType' is not iterable

Once I specified -t, I got pass this error.

Add support for running multiple versions of Docker on CI

We need to be able to run tests on CI with Docker 1.9 and 1.10.

Option --tmp-dir is ignored during squashing

As I can see squash parser has --tmp-dir option, it is parsed but script doesn't pass this argument to Squash class.

Run merging of layers in parallel

If it is possible to run merging of layers in parallel - we should consider this.

Remove squashed layers from Docker daemon

In most cases we're not interested in layers that were squashed. In case where the input and target image has the same tag - all squashed layers will be "orphaned" leaving them in Docker daemon and wasting space. We should probably include a --cleanup switch or something similar to remove layers without tag that were "orphaned" during the squashing process.

Rename project

The main goal of this project is now squashing. The name docker-scripts doesn't really reflect this. Even more - the layers subcommand doesn't make that much sense with new image format. I'm thinking about:

1. Dropping the layers subcommand, and
2. Renaming the project.

This will of course change how CLI is executed and that change will not be backwards compatible. Of course it's fine to do so because we haven't released 1.0.0 yet.

Thoughts?

error when trying to run docker-scripts layers/squash

$ docker-scripts layers myapp/frontend
Error while creating the Docker client: Error while fetching server API version: ('Connection aborted.', error(2, 'No such file or directory'))
Please make sure that you specified valid parameters in the 'DOCKER_CONNECTION' environment variable.

platform osx 10.11
docker version 1.8.1

Performance analysis

In this ticket I want to gather all performance analysis that could make squashing even faster.

All performance related issues in one place: https://github.com/goldmann/docker-scripts/issues?q=is%3Aissue+is%3Aopen+label%3Aperformance

squashing layers removes required `docker_version` key

After squashing layers, the remaining metadata is missing the 'docker_version' key. This should be preserved from the keys in the squashed layers.

Make sure the generated Is is not integer after shortening

https://bugzilla.redhat.com/show_bug.cgi?id=1059122

DOCKER_CONNECTION issue on OSX due to no docker.sock, it uses a VirutalBox VM

There is no docker.sock for use with OSX since it's inside the VM for Docker, if this is still able to work maybe supply a document or note on how to change the value for OSX.

Print version information as part of the execution when verbose is enabled

It would be handy to know the version of the tool.

Getting client incompatibility because of docker_py 1.3.0

Traceback (most recent call last):
  File "/root/.local/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.4.2', 'console_scripts', 'docker-scripts')()
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 91, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 86, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 30, in run_squash
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 48, in __init__
  File "build/bdist.linux-x86_64/egg/docker_scripts/lib/common.py", line 37, in docker_client
  File "build/bdist.linux-x86_64/egg/docker_scripts/lib/common.py", line 52, in valid_docker_connection
  File "/root/.local/lib/python2.7/site-packages/docker_py-1.3.0-py2.7.egg/docker/client.py", line 563, in ping
    return self._result(self._get(self._url('/_ping')))
  File "/root/.local/lib/python2.7/site-packages/docker_py-1.3.0-py2.7.egg/docker/clientbase.py", line 106, in _result
    self._raise_for_status(response)
  File "/root/.local/lib/python2.7/site-packages/docker_py-1.3.0-py2.7.egg/docker/clientbase.py", line 102, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 404 Client Error: Not Found ("client and server don't have same version (client : 1.19, server: 1.18)")

Solution:

$ easy_install --user docker_py==1.2.3 docker-scripts==0.4.2

Make Docker client connection configurable

Investigate if we can change the permissions of files only without duplicating files

Currently if you change the permission of the file only the content of the file is copied and a new layer is created with the same file but with new permissions.

We need to investigate if we can change the permission only without copying the file itself.

Make it possible to run squashing without accessing Docker daemon

It should be possible to run squashing on a provided tar archive containing the image.

check if provided image is actually specified

When provided image is not specified, squash tool ends with this weird traceback. Took me some time to figure out what's actually wrong.

docker-python-1.2.3-3.el7.x86_64
python-docker-scripts-0.4.2

2015-07-14 08:18:32,629 - atomic_reactor.plugin - DEBUG - Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/atomic_reactor-1.3.7-py2.7.egg/atomic_reactor/plugin.py", line 180, in run
    plugin_response = plugin_instance.run()
  File "/usr/lib/python2.7/site-packages/atomic_reactor-1.3.7-py2.7.egg/atomic_reactor/plugins/prepub_squash.py", line 105, in run
    tag=self.tag, output_path=self.workflow.exported_squashed_image.get("path")).run()
  File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 397, in run
    old_image_id = self.docker.inspect_image(self.image)['Id']
  File "/usr/lib/python2.7/site-packages/docker/utils/decorators.py", line 13, in wrapped
    'image or container param is undefined'
NullResource: image or container param is undefined

six 1.3 doesn't have PY2

Version:

python-six-1.3.0-4.el7.noarch

2015-05-13T16:06:17.324907639Z   File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 22, in <module>
2015-05-13T16:06:17.335738340Z     if six.PY2:
2015-05-13T16:06:17.335738340Z AttributeError: 'module' object has no attribute 'PY2'

squash: run could return ID of the squashed image

so dock can consume it

Cleanup temporary directory when "no space left on device" error occurs

When I run docker-scripts and during image squashing error occurs, it doesn't remove temporary directory after it fails:

2015-10-08 08:51:10,796 root         INFO     Squashing image 'centos/php-56-centos7'...
2015-10-08 08:51:10,873 root         INFO     Old image has 17 layers
2015-10-08 08:51:10,873 root         INFO     Attempting to squash from layer 8b2142fa4463437efd3a3460d8a1d13d6d841cd56e9b89d48d12892956e985c4...
2015-10-08 08:51:10,873 root         INFO     Checking if squashing is necessary...
2015-10-08 08:51:10,873 root         INFO     We have 11 layers to squash
2015-10-08 08:51:10,875 root         INFO     Saving image f145f0473731f303f6a3479339a1f18133e043c9486d46f11e40374a76759d8f to /tmp/docker-squash-n3ST8m/image.tar file...
2015-10-08 08:51:34,122 root         INFO     Image saved!
2015-10-08 08:51:34,217 root         INFO     Unpacking /tmp/docker-squash-n3ST8m/image.tar tar file to /tmp/docker-squash-n3ST8m/old directory
Traceback (most recent call last):
  File "/home/vagrant/.local/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.4.2', 'console_scripts', 'docker-scripts')()
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 91, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 86, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 30, in run_squash
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 471, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 91, in _unpack
  File "/usr/lib64/python2.7/tarfile.py", line 2051, in extractall
    self.extract(tarinfo, path)
  File "/usr/lib64/python2.7/tarfile.py", line 2088, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name))
  File "/usr/lib64/python2.7/tarfile.py", line 2164, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib64/python2.7/tarfile.py", line 2205, in makefile
    copyfileobj(source, target)
  File "/usr/lib64/python2.7/tarfile.py", line 265, in copyfileobj
    shutil.copyfileobj(src, dst)
  File "/usr/lib64/python2.7/shutil.py", line 52, in copyfileobj
    fdst.write(buf)
IOError: [Errno 28] No space left on device
...
$ ll /tmp/
total 0
drwx------. 3 vagrant vagrant 80 Oct  8 08:51 docker-squash-n3ST8m

(docker-v110) Fails with mixed-layer image

I'm using the docker-v110 branch. When building an image with docker 1.10 that inherits from an image built with docker < 1.10, squashing fails.

# docker build --rm -t mixed .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM busybox
 ---> 3240943c9ea3
Step 2 : CMD /bin/env
 ---> Running in 4dc560c8c72b
 ---> f3f397aee4b0
Removing intermediate container 4dc560c8c72b
Successfully built f3f397aee4b0

# docker history mixed
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
f3f397aee4b0        2 minutes ago       /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/e   0 B                 
3240943c9ea3        8 days ago          /bin/sh -c #(nop) CMD ["sh"]                    0 B                 
<missing>           8 days ago          /bin/sh -c #(nop) ADD file:7cdf7a89f6a004b2e9   1.114 MB            

# docker-scripts squash mixed -t mixed:squashed
2016-02-25 11:14:05,730 root         INFO     docker-scripts version 1.0.0a1, Docker c3959b1, API 1.22...
2016-02-25 11:14:05,731 root         INFO     Using v2 image format
2016-02-25 11:14:05,738 root         ERROR    Could not get the layer ID to squash, please check provided 'layer' argument: <missing>
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 89, in run
    args.func(args)
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 32, in run_squash
    from_layer=args.from_layer, tag=args.tag, output_path=args.output_path, tmp_dir=args.tmp_dir).run()
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 68, in run
    new_image_id = image.squash()
  File "build/bdist.linux-x86_64/egg/docker_scripts/image.py", line 65, in squash
    self._before_squashing()
  File "build/bdist.linux-x86_64/egg/docker_scripts/v2_image.py", line 15, in _before_squashing
    super(V2Image, self)._before_squashing()
  File "build/bdist.linux-x86_64/egg/docker_scripts/image.py", line 130, in _before_squashing
    "Could not get the layer ID to squash, please check provided 'layer' argument: %s" % from_layer)
SquashError: Could not get the layer ID to squash, please check provided 'layer' argument: <missing>

squash: missing file from ADD instruction

$ docker images
REPOSITORY                                                TAG                  IMAGE ID            CREATED             VIRTUAL SIZE
new-rsyslog2                                              latest               7c7ca4bdd3a6        14 minutes ago      154.1 MB
new-rsyslog                                               latest               85b99087cea9        15 minutes ago      308.2 MB
rsyslog                                                   latest               28206e3d4acd        13 days ago         211.1 MB

$ docker run --rm 28206e3d4acd ls -lhaZ /bin/install.sh
-rwxr-xr-x. root root system_u:object_r:unlabeled_t:s0 /bin/install.sh

$ docker run --rm new-rsyslog2 ls -lhaZ /bin/install.sh
ls: cannot access /bin/install.sh: No such file or directory

$ docker run --rm new-rsyslog ls -lhaZ /bin/install.sh
ls: cannot access /bin/install.sh: No such file or directory

Bump docker client timeout value and make it configurable

Marker files are not added back when we squash layers that modify and remove files

Do not create squashed tar in-memory, store it on disk

When the build is executed on a host that has small amount of RAM and/or the image we squash is big enough - sometimes the tool can eat whole memory and fail with a MemoryError.

Add basic documentation

Things that needs to be covered:

• available environment variables

Do not add back marker files when files do not exist in the unsquashed layers

The problem is that we do not scan for images layers that are not meant to be squashed. If we would do so - we would know if the file that is meant to be hidden (marker file points to it) exists in the layers we do not squash. If it does, we need to leave the marker files so the file will be hidden. But if it does not exist there - we don't need to add this file back.

pulp raises exception when trying to import squashed image

  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/async/tasks.py", line 316, in __call__
    return super(Task, self).__call__(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/pulp/server/managers/content/upload.py", line 225, in import_uploaded_unit
    unit_metadata, file_path, conduit, call_config)
  File "/usr/lib/python2.6/site-packages/pulp_docker/plugins/importers/importer.py", line 139, in upload_unit
    upload.save_models(conduit, models, ancestry, file_path)
  File "/usr/lib/python2.6/site-packages/pulp_docker/plugins/importers/upload.py", line 86, in save_models
    json_dest.write(archive.extractfile(json_src_path).read())
  File "/usr/lib64/python2.6/tarfile.py", line 2100, in extractfile
    tarinfo = self.getmember(member)
  File "/usr/lib64/python2.6/tarfile.py", line 1788, in getmember
    raise KeyError("filename %r not found" % name)

I've tried importing non-squashed image and that worked. Not sure what's really going (b/c I don't have access to server & logs). @mhrivnak, could you please help us here?

Marker files should remove files from all layers

When a marker file for some file is created and the file modified in the same layer - it'll remove the modification only, but the file will not be hidden from previous layers.

unicode error squashing an image

2015-05-12 18:06:00,691 root         INFO     Squashing image 'openshift/perl-516-centos7-candidate'...
2015-05-12 18:06:00,748 root         INFO     Old image has 13 layers
2015-05-12 18:06:00,748 root         INFO     Attempting to squash from layer a0dd78b24d0bb9f011aaf5ab26adc5e07c7d12c2bb796622fd17e46f8632b13b...
2015-05-12 18:06:00,748 root         INFO     Checking if squashing is necessary...
2015-05-12 18:06:00,748 root         INFO     We have 9 layers to squash
2015-05-12 18:06:00,749 root         INFO     Saving image 8fc76b1b4170a0ec291f0f8764d6fda6eec49b35478f668863ffd5929f298b59 to /tmp/tmp-docker-squash-PzH66I/image.tar file...
2015-05-12 18:06:24,545 root         INFO     Image saved!
2015-05-12 18:06:24,547 root         INFO     Unpacking /tmp/tmp-docker-squash-PzH66I/image.tar tar file to /tmp/tmp-docker-squash-PzH66I/old directory
2015-05-12 18:06:27,427 root         INFO     Archive unpacked!
2015-05-12 18:06:27,427 root         INFO     New layer ID for squashed content will be: 60447d4df47226abcc8224550cfa49582f2b2e400552e83167b2cac2f3d7726a
2015-05-12 18:06:27,428 root         INFO     Starting squashing...
2015-05-12 18:06:27,428 root         INFO     Squashing layer 8fc76b1b4170a0ec291f0f8764d6fda6eec49b35478f668863ffd5929f298b59...
2015-05-12 18:06:27,428 root         INFO     Squashing layer 3be96d4576bfda0d8c4f1a64f5e8b6fac537f65c321909850f8897afe833de15...
2015-05-12 18:06:27,428 root         INFO     Squashing layer 77e7705293eb903586e61f194329689d612389dd59026fe0c906c2133ee67c65...
Traceback (most recent call last):
  File "/root/.local/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.3.2', 'console_scripts', 'docker-scripts')()
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 87, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 82, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 30, in run_squash
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 411, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 245, in _squash_layers
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 124, in _marker_files
  File "/usr/lib64/python2.7/tarfile.py", line 1805, in getmembers
    self._load()        # all members, we first have to
  File "/usr/lib64/python2.7/tarfile.py", line 2380, in _load
    tarinfo = self.next()
  File "/usr/lib64/python2.7/tarfile.py", line 2319, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/usr/lib64/python2.7/tarfile.py", line 1242, in fromtarfile
    return obj._proc_member(tarfile)
  File "/usr/lib64/python2.7/tarfile.py", line 1264, in _proc_member
    return self._proc_pax(tarfile)
  File "/usr/lib64/python2.7/tarfile.py", line 1394, in _proc_pax
    value = value.decode("utf8")
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 4: invalid start byte

you can reproduce this by cloning github.com/openshift/sti-perl and running "make test"

Make output parseable

Sometimes it would be good to have the output parseable so other tools could benefit from it, even if not written in Python.

What format it should take? Or maybe it should just output the plain messages?

/cc @bparees

Intended use for multiple RUN commands

I have the following Dockerfiles:

# small
FROM ubuntu:14.04
RUN fallocate -l 100MB a.foo \
 && fallocate -l 150MB b.foo \
 && rm a.foo \
 && rm b.foo

# large
FROM ubuntu:14.04
RUN fallocate -l 100MB a.foo
RUN fallocate -l 150MB b.foo
RUN rm a.foo
RUN rm b.foo

I tried running docker-squash -t squashed large and but got errors regarding missing argument:

SquashError: Could not get the layer ID to squash, please check provided 'layer' argument: <missing>

I installed the docker-v110 version.

Fix logging image ID

New squashed image ID is None

squash fails with KeyError: "linkname '' not found"

I have test-hello image created from this Dockerfile.

# docker-scripts layers -t test-hello
48ecf305d2cf7046c1f5f8fcbcd4994403173441d4a7f125b1bb0ceead9de731
 └─ ded7cd95e059788f2586a51c275a4f151653779d6a7f4dad77c2bd34601d94e4 [u'docker.io/fedora:latest', u'fedora:latest']
  └─ 174a79e36363cf9d536c1ab63c517bdd4d6b6641135e0a7a8d1e4fd06cf152ef [u'test-hello:latest']

When I run
# docker-scripts squash -t sqtest test-hello
I see

2015-08-31 17:45:46,546 root         INFO     Squashing image 'test-hello'...
2015-08-31 17:45:46,554 root         INFO     Old image has 3 layers
2015-08-31 17:45:46,554 root         INFO     Attempting to squash from layer 48ecf305d2cf7046c1f5f8fcbcd4994403173441d4a7f125b1bb0ceead9de731...
2015-08-31 17:45:46,554 root         INFO     Checking if squashing is necessary...
2015-08-31 17:45:46,554 root         INFO     We have 2 layers to squash
2015-08-31 17:45:46,556 root         INFO     Saving image 174a79e36363cf9d536c1ab63c517bdd4d6b6641135e0a7a8d1e4fd06cf152ef to /tmp/docker-squash-xtRB2H/image.tar file...
2015-08-31 17:45:47,935 root         INFO     Image saved!
2015-08-31 17:45:47,942 root         INFO     Unpacking /tmp/docker-squash-xtRB2H/image.tar tar file to /tmp/docker-squash-xtRB2H/old directory
2015-08-31 17:45:48,097 root         INFO     Archive unpacked!
2015-08-31 17:45:48,097 root         INFO     New layer ID for squashed content will be: 7213f8f7af304341d89c08b1554aa030207f3bd07782363133e6a36b1917e756
2015-08-31 17:45:48,097 root         INFO     Starting squashing...
2015-08-31 17:45:48,097 root         INFO     Squashing layer '174a79e36363cf9d536c1ab63c517bdd4d6b6641135e0a7a8d1e4fd06cf152ef'...
2015-08-31 17:45:48,098 root         INFO     Squashing layer 'ded7cd95e059788f2586a51c275a4f151653779d6a7f4dad77c2bd34601d94e4'...
Traceback (most recent call last):
  File "/usr/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.4.2', 'console_scripts', 'docker-scripts')()
  File "/usr/lib/python2.7/site-packages/docker_scripts/cli.py", line 91, in run
    cli.run()
  File "/usr/lib/python2.7/site-packages/docker_scripts/cli.py", line 86, in run
    args.func(args)
  File "/usr/lib/python2.7/site-packages/docker_scripts/cli.py", line 30, in run_squash
    from_layer=args.from_layer, tag=args.tag, output_path=args.output_path).run()
  File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 493, in run
    layers_to_squash, layers_to_move, squashed_tar, old_image_dir)
  File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 384, in _squash_layers
    member, layer_tar.extractfile(member))
  File "/usr/lib64/python2.7/tarfile.py", line 2156, in extractfile
    return self.extractfile(self._find_link_target(tarinfo))
  File "/usr/lib64/python2.7/tarfile.py", line 2435, in _find_link_target
    raise KeyError("linkname %r not found" % linkname)
KeyError: "linkname '' not found"

This is with 0.4.2, but I tried also 0.4.3 from git repo.

python-docker-scripts-0.4.2-2.fc23.noarch
docker-1.8.1-1.git3c1d7c8.fc23.x86_64
python-libs-2.7.10-4.fc23.x86_64

squash: purge working directory when user interrupts squashing with ^C

/var/lib/yum/rpmdb-indexes/.wh.conflicts /var/lib/yum/rpmdb-indexes/.wh.version: no such file or directory

When building the sti-base at the end of build process we squash the images like that and I'm getting following error:

2015-05-22 18:38:08,763 root         INFO     Loading squashed image...
Traceback (most recent call last):
  File "/home/maszulik/.local/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.3.3', 'console_scripts', 'docker-scripts')()
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 87, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 82, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/cli.py", line 30, in run_squash
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 432, in run
  File "build/bdist.linux-x86_64/egg/docker_scripts/squash.py", line 191, in _load_image
  File "/home/maszulik/.local/lib/python2.7/site-packages/docker_py-1.2.2-py2.7.egg/docker/client.py", line 769, in load_image
    self._raise_for_status(res)
  File "/home/maszulik/.local/lib/python2.7/site-packages/docker_py-1.2.2-py2.7.egg/docker/client.py", line 122, in _raise_for_status
    raise errors.APIError(e, response, explanation=explanation)
docker.errors.APIError: 404 Client Error: Not Found ("Untar exit status 1 link /var/lib/yum/rpmdb-indexes/.wh.conflicts /var/lib/yum/rpmdb-indexes/.wh.version: no such file or directory")
Makefile:11: recipe for target 'build' failed
make: *** [build] Error 1

Any ideas?

Squashing fails with "ApplyLayer exit status 1 stdout: stderr: no such file or directory" or similar

Log:

2016-02-04 13:43:43,744 root         DEBUG    Moving unmodified layer '1962253fc10cf35c44d0990bc68291c6b32e8d5c1bda66e4dee2370b2a00c8af'...
2016-02-04 13:43:43,744 root         DEBUG    Moving unmodified layer 'af5068b1ae3eb2e20cf9d94ebcc6df1691725339243262eeb61b48a0a77abc67'...
2016-02-04 13:43:43,745 root         INFO     Squashed image name: squashed:latest
2016-02-04 13:43:43,745 root         DEBUG    Generating tar archive for the squashed image...
2016-02-04 13:43:44,243 root         DEBUG    Archive generated
2016-02-04 13:43:44,243 root         DEBUG    Loading squashed image...
2016-02-04 13:43:51,325 requests.packages.urllib3.connectionpool DEBUG    "POST /v1.21/images/load HTTP/1.1" 404 105
Traceback (most recent call last):
  File "/home/goldmann/.local/bin/docker-scripts", line 9, in <module>
    load_entry_point('docker-scripts==0.4.4', 'console_scripts', 'docker-scripts')()
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker_scripts/cli.py", line 96, in run
    cli.run()
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker_scripts/cli.py", line 88, in run
    args.func(args)
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker_scripts/cli.py", line 31, in run_squash
    from_layer=args.from_layer, tag=args.tag, output_path=args.output_path, tmp_dir=args.tmp_dir).run()
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker_scripts/squash.py", line 520, in run
    self._load_image(new_image_dir)
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker_scripts/squash.py", line 223, in _load_image
    self.docker.load_image(buf.getvalue())
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker/api/image.py", line 148, in load_image
    self._raise_for_status(res)
  File "/home/goldmann/.local/lib/python2.7/site-packages/docker/client.py", line 141, in _raise_for_status
    raise errors.NotFound(e, response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("ApplyLayer exit status 1 stdout:  stderr: chtimes /opt/webserver/webapps/ROOT: no such file or directory")

Docker daemon log:

Feb 04 13:43:45 mistress docker[396]: time="2016-02-04T13:43:45.162110621+01:00" level=debug msg="Calling POST /v1.21/images/load"
Feb 04 13:43:45 mistress docker[396]: time="2016-02-04T13:43:45.162138096+01:00" level=debug msg="POST /v1.21/images/load"
Feb 04 13:43:45 mistress docker[396]: time="2016-02-04T13:43:45.162935119+01:00" level=info msg="{Action=load, Username=goldmann, LoginUID=1000, PID=634}"
Feb 04 13:43:46 mistress docker[396]: time="2016-02-04T13:43:46.260096970+01:00" level=debug msg="Loading d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe"
Feb 04 13:43:46 mistress docker[396]: time="2016-02-04T13:43:46.765199610+01:00" level=debug msg="[deviceset] AddDevice(hash=d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe basehash=af5068b1ae3eb2e20cf9d94ebcc6df1691725339243262eeb61b48a0a77abc67)"
Feb 04 13:43:46 mistress docker[396]: time="2016-02-04T13:43:46.780599317+01:00" level=debug msg="registerDevice(4678, d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:46 mistress docker[396]: time="2016-02-04T13:43:46.793644342+01:00" level=debug msg="[deviceset] AddDevice(hash=d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe basehash=af5068b1ae3eb2e20cf9d94ebcc6df1691725339243262eeb61b48a0a77abc67) END"
Feb 04 13:43:46 mistress docker[396]: time="2016-02-04T13:43:46.795614828+01:00" level=debug msg="activateDeviceIfNeeded(d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:46 mistress docker[396]: time="2016-02-04T13:43:46.848969915+01:00" level=debug msg="Start untar layer"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.007371332+01:00" level=debug msg="[devmapper] UnmountDevice(hash=d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.007400305+01:00" level=debug msg="[devmapper] Unmount(/var/lib/docker/devicemapper/mnt/d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.418713471+01:00" level=debug msg="[devmapper] Unmount done"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.418811361+01:00" level=debug msg="[devmapper] deactivateDevice(d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.418936583+01:00" level=debug msg="[devmapper] removeDevice START(docker-253:4-5242881-d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.440845525+01:00" level=debug msg="[devmapper] removeDevice END(docker-253:4-5242881-d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.440878944+01:00" level=debug msg="[devmapper] deactivateDevice END(d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.440889454+01:00" level=debug msg="[devmapper] UnmountDevice(hash=d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe) END"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.442179582+01:00" level=debug msg="devmapper: DeleteDevice(hash=d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe syncDelete=false) START"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.442192741+01:00" level=debug msg="devmapper: issueDiscard(device: d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe). START"
Feb 04 13:43:49 mistress docker[396]: time="2016-02-04T13:43:49.442200543+01:00" level=debug msg="activateDeviceIfNeeded(d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.271166347+01:00" level=debug msg="devmapper: issueDiscard(device: d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe). END"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.271200577+01:00" level=debug msg="[devmapper] deactivateDevice(d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.271260569+01:00" level=debug msg="[devmapper] removeDevice START(docker-253:4-5242881-d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.296791905+01:00" level=debug msg="[devmapper] removeDevice END(docker-253:4-5242881-d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.296825740+01:00" level=debug msg="[devmapper] deactivateDevice END(d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.317886433+01:00" level=debug msg="unregisterDevice(4678, d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe)"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.320975273+01:00" level=debug msg="devmapper: DeleteDevice(hash=d500bc3c8bd13630b7536b2116cd0a68de1172876c0befb295c3314df66f2cfe syncDelete=false) END"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.321682713+01:00" level=error msg="Handler for POST /v1.21/images/load returned error: ApplyLayer exit status 1 stdout:  stderr: chtimes /opt/webserver/webapps/ROOT: no such file or directory"
Feb 04 13:43:51 mistress docker[396]: time="2016-02-04T13:43:51.322406201+01:00" level=error msg="HTTP Error" err="ApplyLayer exit status 1 stdout:  stderr: chtimes /opt/webserver/webapps/ROOT: no such file or directory" statusCode=404

change ordering of squaching layer

latest layer - the most recent layer
oldest layer - the first layer (base image)

if you will change the way of squashing layers to from latest to oldest you can add only files which are not in squashed layer, because in the case of duplicity you already have the correct one.
This will reduce time for squashing (no need to add duplicates and no need to remove them)

Squashing restore removed files from base layers

See this:

http://fpaste.org/275373/44281144

The first build is without squash. I have one step that removes the "tar" binary and then when the image is commited the tar is really gone.
The second build is with squashing. The same Dockerfile that removes the "tar", but after the image is squashed, the "tar" is back.

The "tar" is added by the base image, so I guess when you squash, it gets restored from the base image layer.

This might be potential security problem if you removing secrets in upper layers.

Incompatibility with old python-six package: AttributeError: '_MovedItems' object has no attribute 'range'

2015-05-20T06:20:54.692561024Z 2015-05-20 02:20:54,688 - dock.plugin - DEBUG - Traceback (most recent call last):
2015-05-20T06:20:54.692561024Z   File "/usr/lib/python2.7/site-packages/dock/plugin.py", line 211, in run
2015-05-20T06:20:54.692561024Z     plugin_response = plugin_instance.run()
2015-05-20T06:20:54.692561024Z   File "/usr/lib/python2.7/site-packages/dock/plugins/prepub_squash.py", line 58, in run
2015-05-20T06:20:54.692561024Z     from_layer=self.from_layer, tag=self.tag).run()
2015-05-20T06:20:54.692561024Z   File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 388, in run
2015-05-20T06:20:54.692561024Z     if not self._save_image(old_image_id, old_image_tar):
2015-05-20T06:20:54.692561024Z   File "/usr/lib/python2.7/site-packages/docker_scripts/squash.py", line 66, in _save_image
2015-05-20T06:20:54.692561024Z     for x in six.moves.range(3):
2015-05-20T06:20:54.692561024Z AttributeError: '_MovedItems' object has no attribute 'range'

(docker-v110) Squashes too many layers

Here, I'm trying to squash a 4-layer image into a 3-layer image. Am I doing it wrong?

# docker pull fedora
Using default tag: latest
latest: Pulling from library/fedora
a3ed95caeb02: Already exists 
236608c7b546: Already exists 
Digest: sha256:1fa98be10c550ffabde65246ed2df16be28dc896d6e370dab56b98460bd27823
Status: Image is up to date for fedora:latest
[root@osbs test]# cat Dockerfile 
FROM fedora
CMD /bin/env
LABEL foo=bar
# docker build -t fedora-test:unsquashed --rm .
Sending build context to Docker daemon 2.048 kB
Step 1 : FROM fedora
 ---> ddd5c9c1d0f2
Step 2 : CMD /bin/env
 ---> Running in 295ae8a4d526
 ---> df9cd69cbf2f
Removing intermediate container 295ae8a4d526
Step 3 : LABEL foo bar
 ---> Running in 8069ddafe169
 ---> dec73734cc6b
Removing intermediate container 8069ddafe169
Successfully built dec73734cc6b
# docker history fedora-test:unsquashed
IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
dec73734cc6b        56 seconds ago       /bin/sh -c #(nop) LABEL foo=bar                 0 B                 
df9cd69cbf2f        About a minute ago   /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/e   0 B                 
ddd5c9c1d0f2        2 days ago           /bin/sh -c #(nop) ADD file:bcb5e5cddd4c4d1cac   204.7 MB            
<missing>           2 days ago           /bin/sh -c #(nop) MAINTAINER Patrick Uiterwij   0 B                 
# docker inspect ddd5c9c1d0f2 | grep -w Id
        "Id": "sha256:ddd5c9c1d0f2a08c5d53958a2590495d4f8a6166e2c1331380178af425ac9f3c",
# docker-scripts -v squash fedora-test:unsquashed -t fedora-test:squashed -f sha256:ddd5c9c1d0f2a08c5d53958a2590495d4f8a6166e2c1331380178af425ac9f3c
2016-03-07 12:31:04,916 root         DEBUG    Running version 1.0.0a1
2016-03-07 12:31:04,916 docker.auth.auth DEBUG    File doesn't exist
2016-03-07 12:31:04,925 requests.packages.urllib3.connectionpool DEBUG    "GET /version HTTP/1.1" 200 208
2016-03-07 12:31:04,926 requests.packages.urllib3.connectionpool DEBUG    "GET /v1.22/_ping HTTP/1.1" 200 2
2016-03-07 12:31:04,928 requests.packages.urllib3.connectionpool DEBUG    "GET /v1.22/version HTTP/1.1" 200 208
2016-03-07 12:31:04,929 root         INFO     docker-scripts version 1.0.0a1, Docker c3959b1, API 1.22...
2016-03-07 12:31:04,929 root         INFO     Using v2 image format
2016-03-07 12:31:04,929 root         DEBUG    Using /tmp/docker-squash-KF64AW as the temporary directory
2016-03-07 12:31:04,933 requests.packages.urllib3.connectionpool DEBUG    "GET /v1.22/images/fedora-test%3Aunsquashed/json HTTP/1.1" 200 1583
2016-03-07 12:31:04,936 requests.packages.urllib3.connectionpool DEBUG    "GET /v1.22/images/sha256%3Adec73734cc6b06b4dd0d1f744364780f808fd863452c87a294f138d2436ce788/history HTTP/1.1" 200 858
2016-03-07 12:31:04,938 requests.packages.urllib3.connectionpool DEBUG    "GET /v1.22/images/sha256%3Addd5c9c1d0f2a08c5d53958a2590495d4f8a6166e2c1331380178af425ac9f3c/json HTTP/1.1" 200 1431
2016-03-07 12:31:04,939 root         INFO     Old image has 4 layers
2016-03-07 12:31:04,939 root         DEBUG    Old layers: [u'<missing>', u'sha256:ddd5c9c1d0f2a08c5d53958a2590495d4f8a6166e2c1331380178af425ac9f3c', u'sha256:df9cd69cbf2ff56b869dd091668e7600cd4657f5043a38f638ce52e7d41a743a', u'sha256:dec73734cc6b06b4dd0d1f744364780f808fd863452c87a294f138d2436ce788']
2016-03-07 12:31:04,939 root         INFO     Checking if squashing is necessary...
2016-03-07 12:31:04,939 root         INFO     Attempting to squash from layer sha256:ddd5c9c1d0f2a08c5d53958a2590495d4f8a6166e2c1331380178af425ac9f3c...
2016-03-07 12:31:04,939 root         INFO     We have 2 layers to squash
2016-03-07 12:31:04,939 root         DEBUG    Layers to squash: [u'sha256:df9cd69cbf2ff56b869dd091668e7600cd4657f5043a38f638ce52e7d41a743a', u'sha256:dec73734cc6b06b4dd0d1f744364780f808fd863452c87a294f138d2436ce788']
2016-03-07 12:31:04,939 root         INFO     Saving image sha256:dec73734cc6b06b4dd0d1f744364780f808fd863452c87a294f138d2436ce788 to /tmp/docker-squash-KF64AW/old/image.tar file...
2016-03-07 12:31:04,939 root         DEBUG    Try #1...
2016-03-07 12:31:09,942 requests.packages.urllib3.connectionpool DEBUG    "GET /v1.22/images/sha256%3Adec73734cc6b06b4dd0d1f744364780f808fd863452c87a294f138d2436ce788/get HTTP/1.1" 200 None
2016-03-07 12:31:13,601 root         INFO     Image saved!
2016-03-07 12:31:13,633 root         INFO     Unpacking /tmp/docker-squash-KF64AW/old/image.tar tar file to /tmp/docker-squash-KF64AW/old directory
2016-03-07 12:31:13,913 root         DEBUG    Removing exported tar (/tmp/docker-squash-KF64AW/old/image.tar)...
2016-03-07 12:31:14,012 root         INFO     Archive unpacked!
2016-03-07 12:31:14,013 root         INFO     Squashing image 'fedora-test:unsquashed'...
2016-03-07 12:31:14,013 root         DEBUG    Reading '/tmp/docker-squash-KF64AW/old/manifest.json' JSON file...
2016-03-07 12:31:14,013 root         DEBUG    Reading '/tmp/docker-squash-KF64AW/old/dec73734cc6b06b4dd0d1f744364780f808fd863452c87a294f138d2436ce788.json' JSON file...
2016-03-07 12:31:15,335 root         DEBUG    Moving unmodified layer '768d4f50f65f00831244703e57f64134771289e3de919a576441c9140e037ea2'...
2016-03-07 12:31:15,335 root         INFO     New squashed image ID is 13200ac0f182d77cc4a778156eb87faaf1b96df89e40097c5f6b06d541578bba
2016-03-07 12:31:15,335 root         DEBUG    Generating tar archive for the squashed image...
2016-03-07 12:31:15,445 root         DEBUG    Archive generated
2016-03-07 12:31:15,445 root         DEBUG    Loading squashed image...
2016-03-07 12:31:18,956 requests.packages.urllib3.connectionpool DEBUG    "POST /v1.22/images/load HTTP/1.1" 200 0
2016-03-07 12:31:18,967 root         DEBUG    Image loaded!
2016-03-07 12:31:19,000 root         INFO     Image registered in Docker daemon as fedora-test:squashed
2016-03-07 12:31:19,000 root         INFO     Done
[root@osbs test]# docker history fedora-test:squashed
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
13200ac0f182        2 days ago          /bin/sh -c #(nop) ADD file:bcb5e5cddd4c4d1cac   204.7 MB            
<missing>           2 days ago          /bin/sh -c #(nop) MAINTAINER Patrick Uiterwij   0 B                 
# docker version
Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 22:37:33 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 22:37:33 2016
 OS/Arch:      linux/amd64

JSON metadata size should start with upper case

According to: https://github.com/docker/docker/blob/master/image/spec/v1.md#image-json-field-descriptions we should use Size not size.