asottile/dockerfile

Parse a dockerfile into a high-level representation using the official go parser

dockerfile's Issues

Windows wheels ABI tag is (probably) too restrictive

Hi again and thank you very much for providing windows wheels.

There's a small issue - the wheel is tagged as dockerfile-3.3.0-cp38-cp38-win_amd64.whl, the second cp38 is an ABI tag that restricts the wheel to python 3.8 only, so pip on other versions refuses to install this wheel and attempts to build from source anyway. You can reproduce with:

PS> pip --version
pip 23.2.1 from [...] (python 3.11)

PS> pip install --only-binary=:all: dockerfile
ERROR: Could not find a version that satisfies the requirement dockerfile (from versions: none)
ERROR: No matching distribution found for dockerfile

However, when I manually downloaded the wheel and renamed it to dockerfile-3.3.0-cp38-abi3-win_amd64.whl, it installed and worked perfectly well, so it's likely just mislabeled.

Could you please take a look?

'\' is stripped from original

Is it because python interprets one \ as an escape character?

Doesn't yet support latest pypy

I'm currently limiting pypy to 5.3.1 in .travis.yml

The latest pypy triggers an error like this:

Building wheels for collected packages: dockerfile
  Running setup.py bdist_wheel for dockerfile: started
  Running setup.py bdist_wheel for dockerfile: finished with status 'error'
  Complete output from command /home/travis/gopath/src/github.com/asottile/dockerfile/.tox/pypy/bin/pypy -u -c "import setuptools, tokenize;__file__='/tmp/pip-lRJy3v-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpwvn1yZpip-wheel- --python-tag pp257:
  running bdist_wheel
  running build
  running build_ext
  $ GOPATH=/tmp/tmp7i_pYi go get -d
  /tmp/cctGS3Lw.o: In function `main':
  test.c:(.text+0xa): undefined reference to `f'
  collect2: error: ld returned 1 exit status
  $ CGO_CFLAGS=-I/home/travis/gopath/src/github.com/asottile/dockerfile/.tox/pypy/include CGO_LDFLAGS=-Wl,--unresolved-symbols=ignore-all GOPATH=/tmp/tmp7i_pYi go build -buildmode=c-shared -o /tmp/pip-lRJy3v-build/build/lib.linux-x86_64-2.7/dockerfile.pypy-41.so
  # github.com/asottile/dockerfile/pylib
  ./main.go:56: cannot use C.Py_ssize_t(len(strs)) (type C.Py_ssize_t) as type C.long in argument to _Cfunc_PyTuple_New
  ./main.go:63: cannot use C.Py_ssize_t(i) (type C.Py_ssize_t) as type C.long in argument to _Cfunc_PyTuple_SetItem
  ./main.go:89: cannot use C.Py_ssize_t(len(cmds)) (type C.Py_ssize_t) as type C.long in argument to _Cfunc_PyTuple_New
  ./main.go:122: cannot use C.Py_ssize_t(i) (type C.Py_ssize_t) as type C.long in argument to _Cfunc_PyTuple_SetItem
  Traceback (most recent call last):
    File "<module>", line 1, in <module>
    File "/tmp/pip-lRJy3v-build/setup.py", line 37, in <module>
      setup_requires=['setuptools-golang>=0.2.0'],
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/core.py", line 151, in setup
      dist.run_commands()
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/dist.py", line 953, in run_commands
      self.run_command(cmd)
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/home/travis/gopath/src/github.com/asottile/dockerfile/.tox/pypy/site-packages/wheel/bdist_wheel.py", line 179, in run
      self.run_command('build')
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/cmd.py", line 334, in run_command
      self.distribution.run_command(command)
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/command/build.py", line 127, in run
      self.run_command(cmd_name)
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/cmd.py", line 334, in run_command
      self.distribution.run_command(command)
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/dist.py", line 972, in run_command
      cmd_obj.run()
    File "/home/travis/gopath/src/github.com/asottile/dockerfile/.tox/pypy/site-packages/setuptools/command/build_ext.py", line 75, in run
      _build_ext.run(self)
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/command/build_ext.py", line 350, in run
      self.build_extensions()
    File "/opt/python/pypy-5.7.1/lib-python/2.7/distutils/command/build_ext.py", line 459, in build_extensions
      self.build_extension(ext)
    File "/home/travis/gopath/src/github.com/asottile/dockerfile/.tox/pypy/site-packages/setuptools_golang.py", line 111, in build_extension
      _check_call(cmd_build, cwd=pkg_path, env=env)
    File "/home/travis/gopath/src/github.com/asottile/dockerfile/.tox/pypy/site-packages/setuptools_golang.py", line 66, in _check_call
      subprocess.check_call(cmd, cwd=cwd, env=dict(os.environ, **env))
    File "/opt/python/pypy-5.7.1/lib-python/2.7/subprocess.py", line 186, in check_call
      raise CalledProcessError(retcode, cmd)
  CalledProcessError: Command '(u'go', u'build', u'-buildmode=c-shared', u'-o', '/tmp/pip-lRJy3v-build/build/lib.linux-x86_64-2.7/dockerfile.pypy-41.so')' returned non-zero exit status 2
  
  ----------------------------------------
  Failed building wheel for dockerfile

This stems from go treating long and Py_ssize_t as different types (despite Py_ssize_t being typedefd long)

I'm going to try and submit a patch that makes the function signatures actually Py_ssize_t.

Finer-grained support for `ARG name=value`

First of all, thanks a ton for writing these Python bindings -- they've been a pleasure to use 🙂

I've been using dockerfile on a project of mine, and I noticed that lines of the form ARG name=value are represented as though name=value is a single AST value, rather than an associative pair. For example:

>>> import dockerfile
>>> dockerfile.parse_string("ARG FOO=/foo")[0]
Command(cmd='arg', sub_cmd=None, json=False, original='ARG FOO=/foo', start_line=1, end_line=1, flags=(), value=('FOO=/foo',))

I realize this would be a breaking change and would thus warrant a new major version, but it'd be great if this was represented with a two-tuple of (name, value) instead. I'm not great with Go, but I did a cursory scan of the Dockerfile parser and I think this information is saved somewhere in the AST nodes that this wrapper is encapsulating:

// parse environment like statements. Note that this does *not* handle
// variable interpolation, which will be handled in the evaluator.
func parseNameVal(rest string, key string, d *directives) (*Node, error) {
	// This is kind of tricky because we need to support the old
	// variant:   KEY name value
	// as well as the new one:    KEY name=value ...
	// The trigger to know which one is being used will be whether we hit
	// a space or = first.  space ==> old, "=" ==> new


	words := parseWords(rest, d)
	if len(words) == 0 {
		return nil, nil
	}


	// Old format (KEY name value)
	if !strings.Contains(words[0], "=") {
		parts := reWhitespace.Split(rest, 2)
		if len(parts) < 2 {
			return nil, fmt.Errorf(key + " must have two arguments")
		}
		return newKeyValueNode(parts[0], parts[1]), nil
	}


	var rootNode *Node
	var prevNode *Node
	for _, word := range words {
		if !strings.Contains(word, "=") {
			return nil, fmt.Errorf("Syntax error - can't find = in %q. Must be of the form: name=value", word)
		}


		parts := strings.SplitN(word, "=", 2)
		node := newKeyValueNode(parts[0], parts[1])
		rootNode, prevNode = appendKeyValueNode(node, rootNode, prevNode)
	}


	return rootNode, nil
}

(Permalink: https://github.com/moby/buildkit/blob/1c037fd52f3481aded331ee74bff7cb1b4d99be6/frontend/dockerfile/parser/line_parsers.go#L139-L175)

docker/docker/builder/dockerfile/parser dependency moved

FYI the https://godoc.org/github.com/docker/docker/builder/dockerfile/parser package no longer exists.

https://godoc.org/github.com/asottile/dockerfile?import-graph&hide=1

I the latest version, The `cmd` field of Command is now uppercase after parsing

I can't see anywhere if that was an intentional change, but using version 3.2.0 in Python 3.10.1 on Linux (5.15.14-1-lts #1 SMP Tue, 11 Jan 2022 15:37:45 +0000 x86_64 GNU/Linux) causes parsing of files to have the Command.cmd field to be uppercase.

Here is the parse result in the new version

In [1]: import dockerfile

In [3]: conf = dockerfile.parse_file('Dockerfile')

In [4]: conf
Out[4]: 
(Command(cmd='FROM', sub_cmd=None, json=False, original='FROM ubuntu:16.04', start_line=1, end_line=1, flags=(), value=('ubuntu:16.04',)),
 Command(cmd='WORKDIR', sub_cmd=None, json=False, original='WORKDIR foo/bar', start_line=3, end_line=3, flags=(), value=('foo/bar',)),
 Command(cmd='ADD', sub_cmd=None, json=False, original='ADD foo foo', start_line=4, end_line=4, flags=(), value=('foo', 'foo')))

Where Dockerfile is the following:

FROM ubuntu:16.04

WORKDIR foo/bar
ADD foo foo

In the previous versions, it was always lowercase, and since you also display the commands in dockerfile.all_cmds() as lowercase, I think it is not on purpose.

dockerfile.ParseError does not exist

Trying this:

import dockerfile
raise dockerfile.ParseError

Results in an AttributeError:

AttributeError: module 'dockerfile' has no attribute 'ParseError'

Add Alpine Linux support

Hello,

When using Alpine Linux 3.15 (current stable version), the module installs just fine :

/dockerfile # pip install .
Processing /dockerfile
Building wheels for collected packages: dockerfile
  Building wheel for dockerfile (setup.py) ... done
  Created wheel for dockerfile: filename=dockerfile-3.2.0-cp39-abi3-linux_x86_64.whl size=2024210 sha256=399339190d6c9cdd9aba4685640fe1f18d819bd1699c5962cbd6ae80b298b71f
  Stored in directory: /tmp/pip-ephem-wheel-cache-6q91f7zq/wheels/53/fc/99/f64c1c48b5e0925367c1c474ae3bc62c381a9a3612b0c921ee
Successfully built dockerfile
Installing collected packages: dockerfile
Successfully installed dockerfile-3.2.0

However, the resuling module is not usable:

/dockerfile # python3 -c "import dockerfile"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: Error relocating /usr/lib/python3.9/site-packages/dockerfile.abi3.so: (null): initial-exec TLS resolves to dynamic definition in /usr/lib/python3.9/site-packages/dockerfile.abi3.so

With Alpine Linux Edge (unstable version), importing the module results in a segmentation fault.
Both behaviours are related to missing symbols. Eg. on Alpine Linux Edge :

/dockerfile # ldd /usr/lib/python3.10/site-packages/dockerfile.abi3.so 
	/lib/ld-musl-x86_64.so.1 (0x7f814f07c000)
	libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f814f07c000)
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyObject_CallFunction: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyTuple_SetItem: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyErr_SetString: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyUnicode_FromString: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyLong_FromLong: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyObject_GetAttrString: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyBytes_AsString: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyModule_AddObject: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyImport_ImportModule: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyObject_CallMethod: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: _Py_Dealloc: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: Py_DecRef: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyBool_FromLong: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyModule_Create2: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyObject_SetAttrString: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyUnicode_AsUTF8String: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyArg_ParseTuple: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyTuple_New: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyErr_NewException: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: _Py_NoneStruct: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyExc_ValueError: symbol not found
Error relocating /usr/lib/python3.10/site-packages/dockerfile.abi3.so: PyExc_OSError: symbol not found

Alpine Linux uses musl instead of glibc, which is probably the reason of these failures.

Would it be possible to support Alpine Linux?

Windows support

Has anyone successfully got this working on Windows? If so, what steps were necessary?

Please provide Windows wheels

Hi! If this is not too much trouble, please consider distributing windows wheels along with macos and manylinux ones. We'd love to use this package as a dependency in our project but ease of installation is very important to our users.

Thanks!

Cannot use it in vendor

I tried to use it my project, But it failed,

go build -o kedge ./main.go
# github.com/kedgeproject/kedge/vendor/github.com/asottile/dockerfile
vendor/github.com/asottile/dockerfile/parse.go:54:15: undefined: parser.Directive
vendor/github.com/asottile/dockerfile/parse.go:55:2: undefined: parser.SetEscapeToken
vendor/github.com/asottile/dockerfile/parse.go:55:24: undefined: parser.DefaultEscapeToken
vendor/github.com/asottile/dockerfile/parse.go:56:26: too many arguments in call to parser.Parse

Support for ARM64 Linux Wheels

Hello, are you able to add support in the CI/CD pipeline to build and deploy Linux ARM64 wheels to PyPi?
We have adopted dockerfile in a local pipeline that is being run from linux arm64 containers, but these pipelines fail due to the fact that they need golang to compile the source. This is a dependency that we ideally would not really want

Python wheels fail to build with go 1.16

When running tox with go 1.16, there's an error for a missing go.mod file. I tried generating it with

$ go mod init github.com/asottile/dockerfile
$ go mod tidy

but for some reason the go.mod file wasn't copied to the temp directory.

py36 inst-nodeps: /Users/nmalaguti/git/dockerfile/.tox/.tmp/package/1/dockerfile-3.1.0.zip
ERROR: invocation failed (exit code 1), logfile: /Users/nmalaguti/git/dockerfile/.tox/py36/log/py36-4.log
================================================================ log start =================================================================
Looking in indexes: https://pypip.knewton.net/simple/
Processing ./.tox/.tmp/package/1/dockerfile-3.1.0.zip
Building wheels for collected packages: dockerfile
  Building wheel for dockerfile (setup.py): started
  Building wheel for dockerfile (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /Users/nmalaguti/git/dockerfile/.tox/py36/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py'"'"'; __file__='"'"'/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-wheel-xxissus_
       cwd: /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/
  Complete output (44 lines):
  running bdist_wheel
  running build
  running build_ext
  $ GOPATH=/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/tmpp15mi08g go get -d
  go: go.mod file not found in current directory or any parent directory; see 'go help modules'
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py", line 36, in <module>
      build_golang={'root': 'github.com/asottile/dockerfile'},
    File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
      return distutils.core.setup(**attrs)
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 955, in run_commands
      self.run_command(cmd)
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/wheel/bdist_wheel.py", line 299, in run
      self.run_command('build')
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 974, in run_command
      cmd_obj.run()
    File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 79, in run
      _build_ext.run(self)
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build_ext.py", line 339, in run
      self.build_extensions()
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
      self._build_extensions_serial()
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
      self.build_extension(ext)
    File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools_golang.py", line 155, in build_extension
      _check_call(cmd_get, cwd=pkg_path, env=env)
    File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools_golang.py", line 111, in _check_call
      subprocess.check_call(cmd, cwd=cwd, env=dict(os.environ, **env))
    File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '('go', 'get', '-d')' returned non-zero exit status 1.
  ----------------------------------------
  ERROR: Failed building wheel for dockerfile
  Running setup.py clean for dockerfile
Failed to build dockerfile
Installing collected packages: dockerfile
  Attempting uninstall: dockerfile
    Found existing installation: dockerfile 3.1.0
    Uninstalling dockerfile-3.1.0:
      Successfully uninstalled dockerfile-3.1.0
    Running setup.py install for dockerfile: started
    Running setup.py install for dockerfile: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/nmalaguti/git/dockerfile/.tox/py36/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py'"'"'; __file__='"'"'/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-record-dw0un84b/install-record.txt --single-version-externally-managed --compile --install-headers /Users/nmalaguti/git/dockerfile/.tox/py36/include/site/python3.6/dockerfile
         cwd: /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/
    Complete output (46 lines):
    running install
    running build
    running build_ext
    $ GOPATH=/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/tmp42ybfyl8 go get -d
    go: go.mod file not found in current directory or any parent directory; see 'go help modules'
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py", line 36, in <module>
        build_golang={'root': 'github.com/asottile/dockerfile'},
      File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools/__init__.py", line 153, in setup
        return distutils.core.setup(**attrs)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools/command/build_ext.py", line 79, in run
        _build_ext.run(self)
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build_ext.py", line 448, in build_extensions
        self._build_extensions_serial()
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/distutils/command/build_ext.py", line 473, in _build_extensions_serial
        self.build_extension(ext)
      File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools_golang.py", line 155, in build_extension
        _check_call(cmd_get, cwd=pkg_path, env=env)
      File "/Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/setuptools_golang.py", line 111, in _check_call
        subprocess.check_call(cmd, cwd=cwd, env=dict(os.environ, **env))
      File "/Users/nmalaguti/.pyenv/versions/3.6.9/lib/python3.6/subprocess.py", line 311, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '('go', 'get', '-d')' returned non-zero exit status 1.
    ----------------------------------------
  Rolling back uninstall of dockerfile
  Moving to /Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/dockerfile-3.1.0.dist-info/
   from /Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/~ockerfile-3.1.0.dist-info
  Moving to /Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/dockerfile.abi3.h
   from /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-uninstall-e0y5vowq/dockerfile.abi3.h
  Moving to /Users/nmalaguti/git/dockerfile/.tox/py36/lib/python3.6/site-packages/dockerfile.abi3.so
   from /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-uninstall-e0y5vowq/dockerfile.abi3.so
ERROR: Command errored out with exit status 1: /Users/nmalaguti/git/dockerfile/.tox/py36/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py'"'"'; __file__='"'"'/private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-req-build-cyr5jbkx/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/20/1s3nd7t12wvb0wfxsv0tp9jw0000gn/T/pip-record-dw0un84b/install-record.txt --single-version-externally-managed --compile --install-headers /Users/nmalaguti/git/dockerfile/.tox/py36/include/site/python3.6/dockerfile Check the logs for full command output

Probably a dupe of asottile/setuptools-golang#65

pylib can not parse env field correctly

Empty lines Inconsistency

Example dockerfile:

FROM ubuntu:14.04

MAINTAINER joedoe

RUN apt-get update -y

RUN apt-get install -y wget

RUN apt-get install unzip -y

############# Move initial files
#copy the inital flie
COPY URBANCODE_DEPLOY_6.2.2_EN_EVAL.zip /opt/
WORKDIR /opt/
RUN unzip URBANCODE_DEPLOY_6.2.2_EN_EVAL.zip 

############## Install java
#RUN wget -P /opt/ http://javadl.oracle.com/webapps/download/AutoDL?BundleId=216424
#WORKDIR /opt
#RUN mv AutoDL?BundleId=216424 jre-8u111-linux-x64.tar.gz
#RUN tar xvzf jre-8u111-linux-x64.tar.gz
#RUN update-alternatives --install /usr/bin/java java /opt/jre1.8.0_111/bin/java 2069
#WORKDIR /localVolume/ibm-ucd-install
#ENV JAVA_HOME /opt/jre1.8.0_111
RUN apt-get install -y default-jdk
ENV JAVA_HOME /usr/lib/jvm/default-java

############### Install mysql server
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y mysql-server

############## Get the linked driver for the mysql server working
#copy over the install file to the opt directory for the shared 
RUN wget -P /opt/ https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.40.tar.gz
WORKDIR /opt/
RUN tar xvzf mysql-connector-java-5.1.40.tar.gz

############## Move the connector for mysql connector to the installer
RUN cp /opt/mysql-connector-java-5.1.40/mysql-connector-java-5.1.40-bin.jar /opt/ibm-ucd-install/lib/ext

#############copy ovet the installation files
#remove old
RUN rm /opt/ibm-ucd-install/install.properties
#add new
ADD install.properties /opt/ibm-ucd-install/

#insert a line that waits for the previous line to finish so it isnt busy
RUN sed -i "85isync" /opt/ibm-ucd-install/install-server.sh

############# Start the Urban code install chain 
WORKDIR /opt/ibm-ucd-install

#############Create an account and set up database, and run the installer in the same layer
RUN service mysql start\
&& mysql -u root -e "CREATE USER 'ibm_ucd'@'localhost' IDENTIFIED BY 'password';"\
&& mysql -u root -e  "CREATE DATABASE ibm_ucd character set utf8 collate utf8_bin;"\
&& mysql -u root -e  "GRANT ALL ON ibm_ucd.* TO 'ibm_ucd'@'%' IDENTIFIED BY 'password' WITH GRANT OPTION;" 
#insert back slash on previous line to continue 
#&& ./install-server.sh run << EOF\
#\n\
#/usr\n\
#EOF

############this will create a ibm-ucd folder in the opt directory which you can then run inside
#WORKDIR /opt/ibm-ucd/server/bin
#ENTRYPOINT service mysql start && ./server start && bash

Output of empty lines at the end:

Command(cmd=None, sub_cmd=None, json=False, original=None, start_line=60, end_line=81, flags=(), value=('',)))

What's inconsistent is that lines in between are not being read as None while the lines at the end of the file are.

Support comments, directives, and HEALTHCHECK options

dockerfile doesn't return the result of parsed comment lines from a source Dockerfile. While it might be inconvenient but at least tolerable to work around parsing pure comment lines, there is also the escape directive that is specified via a special comment line, and has a semantic effect on the rest of the file. Ability to parse escape directives is necessary.

I also noticed that a parsed HEALTHCHECK command does not put the option argument into the value part of the returned command, only the CMD part. E.g. this line of a Dockerfile

HEALTHCHECK \
  --interval=3s \
  --timeout=30s \
  --start-period=10s \
  --retries=3 \
  CMD echo 'hello'

parses into this Command:

Command(cmd='healthcheck', sub_cmd=None, json=False, original="HEALTHCHECK --interval=3s --timeout=30s --start-period=10s --retries=3   CMD echo 'hello'", start_line=9, value=('CMD', "echo 'hello'"))

Similarly, the --from option of the COPY command is not captured, exactly in the same manner as theHEALTHCHECK options (this is particularly important for multi-stage builds).

As a side note, I have just begun some work on a separate project that will make use of your dockerfile Python wrapper. Thanks very much for making this ... and let me know how I can contribute if you'd like some help with actively maintaining this project.

Upgrade buildkit version and support heredocs syntax

Support heredocs syntax and add them to the command output to consume it.
Currently instructions with heredocs syntax break the parsing (added undefined commands to the output)
https://docs.docker.com/engine/reference/builder/#here-documents
https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/
Thanks!