arcalot/arcaflow-container-toolkit

ACT - The Arcaflow Container Toolkit

arcaflow-container-toolkit's Issues

latest tag not updated in quay.io

Describe the bug

After updating the kubeconfig plugin main branch with a PR, the new build automation successfully pushed a new container image with a commit tag, but the latest tag has not updated to point to a new build from the latest main code.

https://quay.io/repository/arcalot/arcaflow-plugin-kubeconfig?tab=tags

Containers failed to push to ghcr or quay

Describe the bug

In a build of sysbench from a development branch update, the Carpenter automation failed to push containers to either ghcr or quay.

To reproduce

Currently unsure how to reproduce the failure.

Additional context

Failed build action: https://github.com/arcalot/arcaflow-plugin-sysbench/actions/runs/4416765995/jobs/7741514163#step:6:679

Handle Failed Docker Server Connection

Will it fail if the Docker Server is not running and listening on the default socket?

Originally posted by @janosdebugs in #21 (comment)

Minimize container size

Please describe what you would like to see in this project

Because the carpenter container will be run very frequently in different environments via automation, we should try to minimize the container's size while maximizing its performance.

Readme for first time users

Please describe what you would like to see in this project

I would like to see the current README replaced with a guide for first-time users of Carpenter to publish their own plugin.

Please describe your use case

Plugins in other organizations and/or CI systems.

Simple carpenter setup

Please describe what you would like to see in this project

I would like to see Carpenter to be simplified to the point where a new plugin project can be set up without a template from scratch in less than 5 steps and less than 2 minutes, ideally without additional config files. The usage of Carpenter should be intuitive and should not require reading more than 2 screens of documentation. Carpenter should be able to, at the very least, build a plugin image (if not push it) without specifying any configuration file simply by specifying the image name. Carpenter should detect the project type (based on go.mod, requirements.txt, or pyproject.toml, figure out the correct dependency manager to use) and apply the appropriate Dockerfile automatically. Carpenter should accept the --push flag to push to the registry and prompt the user for credentials if credentials are required but have not been provided.

Please describe your use case

I would like to see non-Arcalot projects that are running on other CI platforms enabled to quickly and easily spin up new plugin projects.

:latest tag is not linking to latest release in non-multi-arch builds

Describe the bug

If multi-arch is not enabled in a workflow, the resulting release build pushed to quay.io does not have the :latest tag linked to the release tag.

Carpenter Image

Please describe what you would like to see in this project

• It would be nice if carpenter was implemented in a way that it can be pulled as an image from github packages and ran instead of compiling each time.
• Make sure carpenter is versioned and up to date

Please describe your use case

• This would save quite a bit of build time and also allow us to version control Carpenter for end-users.

Additional context

There are a few things that would need to change in the project itself. Most of the changes will happen in the workflows and the repository's of the plugins as well in the "how" they implement carpenter

Require Official Python Plugins to be valid Python Packages

Please describe what you would like to see in this project

Ensure that the given plugin is a valid Python plugin.

1. has pyproject.toml
2. has a directory at root or under a root src directory that has the same filename in the pyproject.toml at the pyproject.project.name, or tool.poetry.name, key values.

Please describe your use case

This is to enforce some similarity in the directory structure of our Python plugins, and to facilitate publishing them to a public Python package repository, like pypi.

Custom Namespace Push to Quay

Please describe what you would like to see in this project

• Implement a way for carpenter to receive a custom namespace request to push an image to.
• Currently, it assumes arcalot without the ability to change
• Make sure this can be implemented through a reusable workflow as well

Please describe your use case

This will be needed by a few teams that have a use case for the image to live outside of the arcalot repository in quay.

Enable multi-architecture support; Specifically aarch64

Please describe what you would like to see in this project

ACT should support building plugin container images for multiple architectures. Specifically ARM aarch64 is needed for existing use cases.

Please describe your use case

Automatic plugin builds in aarch64 compatible formats will enable direct use of plugins and workflows in ARM environments.

Additional context

This is critical for the chaos use case.

Development builds are not getting quay expiration dates set

Describe the bug

When container builds for tags other than releases or latest are created, they are supposed to have an expiration date in quay.io set. Currently, those expiration dates don't seem to be getting set.

To reproduce

Push changes to a development branch of a plugin that uses the ACT workflow. This should trigger a container build and push to quay.io, and you should see that the expiration date is not set.

Additional context

https://quay.io/repository/arcalot/arcaflow-plugin-sysbench?tab=tags

Release Build System

Stabilize build system api, and release it as a golang package.

1. • refactor sub-package cmd code to another sub-package
2. • add unit tests to reach at least 60% by line coverage (and maybe modified decision/condition coverage)
3. • add golang style and quality linting
4. • release carpenter as a package (try go releaser)

Integrate arcaflow-docsgen

Please describe what you would like to see in this project

As an optional feature, Carpenter should probably integrate Arcaflow-DocsGen to keep a README.md or similar updated. The feature should be optional and the markdown file should be configurable.

Please describe your use case

Keep the documentation for plugins updated.

Additional context

Arcaflow-DocsGen is early in development and should be considered carefully.

Cannot determine programming language of a python package

Describe the bug

If a Python plugin uses a typical directory structure for a Python package, Carpenter cannot determine the programming language of that plugin, throws an error, and causes the plugin's image build to fail.

To reproduce

$ tree .
.
├── arcaflow_plugin_template_python
│   └── example_plugin.py
├── docker-compose.yaml
├── Dockerfile
├── example.yaml
├── LICENSE
├── poetry.lock
├── pyproject.toml
├── README.md
├── requirements.txt
└── tests
    └── test_example_plugin.py

$ docker build --tag plugin-template .
$ cat example.yaml | docker run --rm -i plugin-template --debug -f -
output_id: success
output_data:
  message: Hello, John Doe!

Building with the plugin image builder

docker run --rm -e=IMAGE_TAG="0.3.0" -e=IMAGE_NAME="arcaflow-plugin-template-python" -v /var/run/docker.sock:/var/run/docker.sock:z -v $HOME/workspace/arcaflow/arcaflow-plugin-template-python:/github/workspace carpenter build -v --build
2022-12-01T21:38:11Z	info		Using config file:/.carpenter.yaml
2022-12-01T21:38:11Z	info		
2022-12-01T21:38:11Z	info		
2022-12-01T21:38:11Z	info		GITHUB_NAMESPACE not set
2022-12-01T21:38:11Z	info		
2022-12-01T21:38:11Z	info		
2022-12-01T21:38:11Z	info		
panic: runtime error: slice bounds out of range [1:0]

goroutine 1 [running]:
go.arcalot.io/imagebuilder/internal/requirements.ImageLanguage({0xc0000c3e00, 0x10, 0x0?})
	/build/internal/requirements/containerfile.go:69 +0x2a5
go.arcalot.io/imagebuilder/internal/requirements.LanguageRequirements({0xc00010f848, 0x11}, {0xc0000c3e00, 0x10, 0x10}, {0xc00002a00b, 0x1f}, {0xc00002800a, 0x5}, {0xa5caa0, ...}, ...)
	/build/internal/requirements/common.go:25 +0x6e
go.arcalot.io/imagebuilder/internal/carpentry.Carpentry(0x0?, 0x0?, {0xa5bfe0, 0xc000328c10}, {{0xc0002eac60, 0x8}, {0xc00002a00b, 0x1f}, {0xc00010f848, 0x11}, ...}, ...)
	/build/internal/carpentry/carpentry.go:31 +0x158
go.arcalot.io/imagebuilder/internal/carpentry.CliCarpentry(0x0?, 0x0?, {0xa5caa0, 0xc0002f7080}, {0x983783, 0x6})
	/build/internal/carpentry/carpentry.go:89 +0x4dc
go.arcalot.io/imagebuilder/cmd.glob..func1(0xd3e9c0?, {0x982d1d?, 0x2?, 0x2?})
	/build/cmd/build.go:25 +0x46
github.com/spf13/cobra.(*Command).execute(0xd3e9c0, {0xc000270e00, 0x2, 0x2})
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:920 +0x847
github.com/spf13/cobra.(*Command).ExecuteC(0xd3eca0)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:1044 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
	/go/pkg/mod/github.com/spf13/[email protected]/command.go:968
go.arcalot.io/imagebuilder/cmd.Execute()
	/build/cmd/root.go:41 +0x25
main.main()
	/build/carpenter.go:9 +0x17

Wrong build instructions in README.md

Describe the bug

The README.md file gives the following instructions for building:

Build the builder

• golang v1.17
• current working directory is this project's root directory

go build build.go

If successful, this will result in the arcaflow-plugin-image-builder executable, and it will be named build in your current working directory.

But when running it on a RHEL 8 system you'll get:

$ go build build.go
no required module provides package build.go; to add it:
	go get build.go

$ go version
go version go1.17.12 linux/amd64

To reproduce

Clone the repository and follow instructions in README.md

verify dockerfile entrypoint is a valid plugin source file

1. start the image with --atp golang plugin sdk
2. try to read its schema.

The Go SDK should be able to handle the communication if needed.

Originally posted by @janosdebugs in #21 (comment)

Optional Requirements

Note for later: we should make these requirements part of a more strict set which is only applied if people want to contribute official plugins. For their own plugins people can label them as they please.

Originally posted by @janosdebugs in #21 (comment)

Unhelpful error message when following documentation

Describe the bug

When following the documentation here, Carpenter outputs a super unhelpful bunch of error messages:

# docker run \
>    --rm \
>    -e=IMAGE_NAME=example-plugin \
>    -e=IMAGE_TAG=latest \
>    --volume /var/run/docker.sock:/var/run/docker.sock:z \
>    --volume $PWD/:/github/workspace \
>    ghcr.io/arcalot/arcaflow-plugin-image-builder:v0.3.0 \
>    build --build
Unable to find image 'ghcr.io/arcalot/arcaflow-plugin-image-builder:v0.3.0' locally
v0.3.0: Pulling from arcalot/arcaflow-plugin-image-builder
e92ac2adfcb0: Pull complete
5199db4ecb15: Pull complete
ff3f6419bc6e: Pull complete
5617940ba71c: Pull complete
541c2799b994: Pull complete
401e9d30e2d8: Pull complete
Digest: sha256:1a8f906f4b27d6a798f4ca18876f6497e43acff328c26bef837f33f24dff68d0
Status: Downloaded newer image for ghcr.io/arcalot/arcaflow-plugin-image-builder:v0.3.0
2023-03-22T10:10:49Z    info            Using config file:/.carpenter.yaml
2023-03-22T10:10:49Z    info            GITHUB_USERNAME not set
2023-03-22T10:10:49Z    info            GITHUB_PASSWORD not set
2023-03-22T10:10:49Z    info            GITHUB_NAMESPACE not set
2023-03-22T10:10:49Z    info             not set
2023-03-22T10:10:49Z    info            Missing credentials for ghcr.io
2023-03-22T10:10:49Z    info            QUAY_USERNAME not set
2023-03-22T10:10:49Z    info            QUAY_PASSWORD not set
2023-03-22T10:10:49Z    info            QUAY_NAMESPACE not set
2023-03-22T10:10:49Z    info            QUAY_CUSTOM_NAMESPACE not set
2023-03-22T10:10:49Z    info            Missing credentials for quay.io
2023-03-22T10:10:49Z    info            Missing README.md
2023-03-22T10:10:49Z    info            Missing Dockerfile
2023-03-22T10:10:49Z    info            Missing a test file
2023-03-22T10:10:49Z    info            Missing Dockerfile
plugin.py
2023/03/22 10:10:49 failed requirements check, not building: example-plugin latest

To reproduce

Run Carpenter in an empty directory with th ecommand line described above.

Expected result

• Carpenter should not output things that look like debug or error messages as info (e.g. GITHUB_USERNAME not set). This is confusing for the user because it is unclear if this is a problem.
• Don't use the wording "missing" if it is an optional parameter.
• When failing the run, Carpenter should explain what the user did wrong and how to fix it.

Additional context

In its current state, Carpenter is not usable for non-Arcalot projects, it is way too hard to use.

action not failing if the image is not built due to missing conditions

Describe the bug

arcaflow-plugin-image-builder action is not failing if the image is not built due to missing conditions.

Example 1

2022-10-12T05:43:41Z	info		Missing pyproject.toml
2022-10-12T05:43:42Z	info		Failed requirements check, not building: arcaflow-plugin-template-python latest

As we expect the action to perform a build, it should fail if the build is skipped on failed requirements check