afdaniele/cpk

Code Packaging toolKit

cpk's Issues

Graceful error with rsync

When rsync fails to sync, suggest the user to use -sm/--sync-mirror instead of showing the Python exception.

CPK fails when there is no git HEAD

cpk|    INFO : CPK - Code Packaging toolKit - v0.1.1
fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/home/ripl/anaconda3/bin/cpk", line 6, in <module>
    run()
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/cli/main.py", line 66, in run
    command.execute(machine, parsed)
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/cli/commands/build.py", line 126, in execute
    project = CPKProject(parsed.workdir, parsed=parsed)
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/project.py", line 46, in __init__
    30: GitRepositoryAdapter(self._path),
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/adapters/git.py", line 13, in __init__
    self._repo = get_repo_info(self._path)
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/utils/git.py", line 16, in get_repo_info
    sha = run_cmd(["git", "--git-dir", f'"{path}"',
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/utils/misc.py", line 17, in run_cmd
    lines = subprocess.check_output(cmd, shell=True).decode("utf-8").split("\n")
  File "/home/ripl/anaconda3/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/ripl/anaconda3/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'git --git-dir "/home/ripl/code/test-project/.git" rev-parse HEAD' returned non-zero exit status 128.

cpk decorate appends "-{arch}" to the image name when the user does not supply a tag

For example, calling the following command with an amd64 architecture

"cpk decorate -m 'maintainer' library/ros:noeitc my-image-cpk"

will create a new image named my-image-cpk-amd64:latest.

The Dockerfile used by cpk expects an image name of the form "name:tag-amd64"

Default network mode

cpk run is always passing a network mode to docker.
This is wrong, only pass one if it is not the default.

Empty repository error

When a cpk project is also a git repository with no HEAD reference, you get this error.

fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Traceback (most recent call last):
  File "/home/afdaniele/.local/bin/cpk", line 6, in <module>
    run()
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/cli/main.py", line 66, in run
    command.execute(machine, parsed)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/cli/commands/build.py", line 126, in execute
    project = CPKProject(parsed.workdir, parsed=parsed)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/project.py", line 46, in __init__
    30: GitRepositoryAdapter(self._path),
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/adapters/git.py", line 13, in __init__
    self._repo = get_repo_info(self._path)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/utils/git.py", line 17, in get_repo_info
    "rev-parse", "HEAD"])[0]
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/utils/misc.py", line 17, in run_cmd
    lines = subprocess.check_output(cmd, shell=True).decode("utf-8").split("\n")
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'git --git-dir "/home/afdaniele/code/whoi/whoi-nui-nlu-server/.git" rev-parse HEAD' returned non-zero exit status 128.

source catkin ws

the catkin workspace in /code does not seem to be automatically sourced by the entrypoint, the file setup.sh takes care of it now, but it should be done by the entrypoint.

Format error

cpk|    INFO : Parameter `arch` not given, will resolve it from the endpoint.
cpk|    INFO : Parameter `arch` automatically set to `amd64`.
cpk|    INFO : Running an image for amd64 on amd64.
cpk|    INFO : Working with an `amd64` image on `amd64`. Multiarch not needed!
Traceback (most recent call last):
  File "/home/afdaniele/.local/bin/cpk", line 6, in <module>
    run()
  File "/home/afdaniele/.local/lib/python3.8/site-packages/cpk/cli/main.py", line 66, in run
    command.execute(machine, parsed)
  File "/home/afdaniele/.local/lib/python3.8/site-packages/cpk/cli/commands/run.py", line 280, in execute
    raise ValueError("You cannot use the option --launcher together with --cmd.")
ValueError: You cannot use the option --launcher together with --cmd.

Invalid Autogenerated Docker Image Name

When running cpk-build the autogenerated docker image name uses the repo owner as the start of the name, and when the repo owner's name is capitalized the build returns the error repository name must be lowercase

Bad owner or permissions on machine conf file

Bad owner or permissions on /home/afdaniele/.cpk/ssh/xavier.conf

Traceback (most recent call last):
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1379, in getresponse
    self.close()
  File "/usr/lib/python3.6/http/client.py", line 969, in close
    sock.close()   # close it manually... there may be other refs
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/transport/sshconn.py", line 102, in close
    self.proc.stdin.flush()
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/afdaniele/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 532, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/home/afdaniele/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1379, in getresponse
    self.close()
  File "/usr/lib/python3.6/http/client.py", line 969, in close
    sock.close()   # close it manually... there may be other refs
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/transport/sshconn.py", line 102, in close
    self.proc.stdin.flush()
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/afdaniele/.local/lib/python3.6/site-packages/requests/sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/afdaniele/.local/bin/cpk", line 6, in <module>
    run()
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/cli/main.py", line 83, in run
    res = command.execute(machine, parsed)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/cli/commands/build.py", line 137, in execute
    CLIEndpointInfoCommand.execute(machine, parsed)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/cli/commands/endpoint/info.py", line 26, in execute
    docker = machine.get_client()
  File "/home/afdaniele/.local/lib/python3.6/site-packages/cpk/machine.py", line 106, in get_client
    return docker.DockerClient(base_url=self.base_url, use_ssh_client=True)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/afdaniele/.local/lib/python3.6/site-packages/docker/api/client.py", line 222, in _retrieve_server_version
    'Error while fetching server API version: {0}'.format(e)
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

CPK is not copying the right skeleton dir

The following block is missing from Dockerfile

# build catkin workspace
RUN catkin build \
    --workspace ${CPK_CODE_DIR}

Gracefully check for paramiko

Docker needs paramiko to perform SSH connections to a Docker Engine.
Docker does not declare paramiko as a dependency, so pip does not install it together with docker.
Currently, an import error is thrown when this happens.
Show a cleaner error message.

Docker Rootless with cpk build

Hey there,

I am trying to get docker rootless working with cpk but am getting the error below when trying to build:

cpk| INFO : CPK - Code Packaging toolKit - v0.5.1
cpk| INFO : Project workspace: /home/aiden/nlimb2

Project:
Name: nlimb2
Tag: master
Version: unreleased
Template:
Name: basic
Version: 1.0
Index: Clean
Path: /home/aiden/nlimb2
URL: (none)
Adapters: fs project.cpk git

---

cpk| INFO : Retrieving info about Docker endpoint...

Docker Endpoint:
Machine: from-environment
Hostname: gpu
Operating System: Ubuntu 20.04.5 LTS
Kernel Version: 5.4.0-135-generic
OSType: linux
Architecture: x86_64
Total Memory: 376.54 GB
CPUs: 1

cpk| INFO : Parameter arch not given, will resolve it from the endpoint.
cpk| INFO : Parameter arch automatically set to amd64.
cpk| INFO : Building an image for amd64 on amd64.
cpk| INFO : Working with an amd64 image on amd64. Multiarch not needed!
cpk| INFO : Found an image with the same name. Using it as cache source.

ERRO[2023-01-24T11:44:51.654773017+11:00] Handler for POST /v1.41/build returned error: Error processing tar file(exit status 1): lchown /.git: invalid argument
cpk| ERROR : An error occurred while building the project image:
: 500 Server Error for http+docker://localhost/v1.41/build?t=docker.io%2Fcbschaff%2Fnlimb2%3Amaster-amd64&q=False&nocache=False&rm=True&forcerm=False&pull=False&buildargs=%7B%22ARCH%22%3A+%22amd64%22%2C+%22NAME%22%3A+%22nlimb2%22%2C+%22DESCRIPTION%22%3A+%22development+repo+for+nliimb%22%2C+%22ORGANIZATION%22%3A+%22cbschaff%22%2C+%22MAINTAINER%22%3A+%22Chip+Schaff%22%2C+%22NCPUS%22%3A+%221%22%7D&labels=%7B%22cpk.label.project.cbschaff.nlimb2.code.vcs%22%3A+%22git%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.version.tag%22%3A+%22master%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.version.head%22%3A+%22ND%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.version.closest%22%3A+%22ND%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.version.sha%22%3A+%22a7d2c8363b53f690a70611532c666bcb6a0ac039%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.vcs.repository%22%3A+%22nlimb2%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.vcs.branch%22%3A+%22master%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.vcs.url%22%3A+%22https%3A%2F%2Fgithub.com%2Fcbschaff%2Fnlimb2%22%2C+%22cpk.label.project.cbschaff.nlimb2.template.name%22%3A+%22basic%22%2C+%22cpk.label.project.cbschaff.nlimb2.template.version%22%3A+%221.0%22%2C+%22cpk.label.project.cbschaff.nlimb2.template.url%22%3A+%22https%3A%2F%2Fgithub.com%2Fafdaniele%2Fcpk-template-full%22%2C+%22cpk.label.project.cbschaff.nlimb2.code.launchers%22%3A+%22viz%2Cdefault%2Crecord%2Ctrain%2Cxorg%22%2C+%22cpk.label.project.cbschaff.nlimb2.time%22%3A+%22ND%22%7D&platform=linux%2Famd64: Internal Server Error ("Error processing tar file(exit status 1): lchown /.git: invalid argument")

I believe this may be due to the set default machine environment:

include/cpk/utils/docker.py
DEFAULT_MACHINE = "unix:///var/run/docker.sock"

Is there a way to update the default machine to stop using docker.sock and our socket instead?

When we run docker rootless it uses a different docker socket:

/var/tmp/xdg_runtime_dir_169337/docker.sock

Lots of `None`s get printed on image push

Running the command cpk build --push produces a lot of None in the console while the image is being pushed.

cpk push seems to hang

race condition on SSH key

running two cpk commands against the same cpk machine creates a race condition in which the first command to terminate will clear the temporary SSH key and the second will have issues performing not-yet authorized actions.

Maybe we should create random temporary keys for different instances of commands.

Passing docker arguments with no cpk run arguments

It looks like passing docker args to cpk run is not possible when no cpk run args are passed.

The following happens in the basic example:

BAD:

$ cpk run -- -e DEBUG=1
cpk|    INFO : CPK - Code Packaging toolKit - v0.2.5
usage: cpk run [-h] [-C WORKDIR] [-H MACHINE] [-a {arm64v8,arm32v7,amd64}] [-f] [-v] [--debug] [-n NAME] [-c CMD] [--pull] [--force-pull] [--no-multiarch] [-M [MOUNT]] [--keep] [-L LAUNCHER] [-A ARGUMENTS] [--runtime RUNTIME] [-X] [-s] [--net NETWORK_MODE] [-d] [--tag TAG]
               [docker_args [docker_args ...]]
cpk run: error: unrecognized arguments: -e

GOOD:

$ cpk run --debug  -- -e DEBUG=1
cpk|    INFO : CPK - Code Packaging toolKit - v0.2.5
cpk|   DEBUG : Argument 'parsed.machine' not set. Creating machine from environment.
cpk|    INFO : Project workspace: /home/afdaniele/code/afdaniele/cpk/examples/basic

... ALL GOOD HERE ...

Bad permissions on SSH host config file

When using a cpk machine, the host config file has 664 permissions, it should have 600 instead.
The error shown is the following:

...
Bad owner or permissions on /home/afdaniele/.cpk/ssh/nui-shore.conf
...

docker exec with launcher

When a launcher uses a Python package that is in a directory added to PYTHON_PATH by the entrypoint script, running docker exec ... LAUNCHER does not work, as the entrypoint is "skipped" and the package not discoverable.

One possible fix would be to have the launchers source the entrypoint script.

Default project name

When running the command

cpk create ./my-project

the string my-project should be the suggested project name in the interaction that follows/

cpk decorate regex issue

When calling cpk decorate on a local image, the prefix "library/" is required.

It would be nice to be able to use the same name that docker uses for images.

Repository index cleanliness check on wrong path

Building a project with -C ./path from a directory containing a repository that is not clean, generates the warning that the index is not clean and -f should be used.

Sourcing "/opt/ros/$ROS_DISTRO/setup.bash" before calling catkin build

In order to find ros packages during catkin build, I needed to source ros' setup.bash before calling catkin build.

To fix this, I edited the catkin build line of the docker file to:

# build catkin workspace
RUN /bin/bash -c "source /opt/ros/$ROS_DISTRO/setup.bash && catkin build
--workspace ${CPK_CODE_DIR}"

I'm not sure if this is the solution you want to go with, but it worked in my case.

CPK Fails when there is no remote.origin

Traceback (most recent call last):
  File "/home/ripl/anaconda3/bin/cpk", line 6, in <module>
    run()
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/cli/main.py", line 66, in run
    command.execute(machine, parsed)
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/cli/commands/build.py", line 126, in execute
    project = CPKProject(parsed.workdir, parsed=parsed)
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/project.py", line 46, in __init__
    30: GitRepositoryAdapter(self._path),
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/adapters/git.py", line 13, in __init__
    self._repo = get_repo_info(self._path)
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/utils/git.py", line 22, in get_repo_info
    origin_url = run_cmd(["git", "--git-dir", f'"{path}"',
  File "/home/ripl/anaconda3/lib/python3.8/site-packages/cpk/utils/misc.py", line 17, in run_cmd
    lines = subprocess.check_output(cmd, shell=True).decode("utf-8").split("\n")
  File "/home/ripl/anaconda3/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/home/ripl/anaconda3/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'git --git-dir "/home/ripl/code/test-project/.git" config --get remote.origin.url' returned non-zero exit status 1.