Giter Site home page Giter Site logo

cnab-to-oci's Introduction

CircleCI Documentation

CNAB to OCI

The intent of CNAB to OCI is to propose a reference implementation for sharing a CNAB using an OCI or Docker registry.

Jump to the example.

Rationale for this approach

Goals:

  • Package the information from a CNAB bundle.json into a format that can be stored in container registries.
  • Require no or only minor changes to the OCI specification.
    • Major changes would take long to get approved.
    • Anything that diverges from the current specification will require coordination with registries to ensure compatibility.
  • Store all container images required for the CNAB in the same repository and reference them from the same manifest.
    • If a user can access the CNAB, they can access all the parts needed to install it.
    • Moving a CNAB from one repository to another is atomic.
  • Ensure that registries can reason over these CNABs.
    • Provide enough information for registries to understand how to present these artifacts.

Non-goals:

  • A perfectly clean solution.
    • The authors acknowledge that there is a tension between getting something working today and the ideal solution.

Selection of OCI index

The CNAB specification references a list of invocation images and a map of other images. An OCI index is already used for handling multiple images so this was seen as a natural fit.

The only disadvantage of the OCI index is its lack of a top-level mechanism for communicating type which may make the artifacts more difficult for registries to understand. The authors propose overcoming this using annotations.

Annotations

Annotations are an optional part of the OCI specification. They can be included in the top-level of an OCI index, at the top-level of a manifest, or as part of a descriptor.

While they are generally unrestricted key-value pairs, some guidance is given for keys by the OCI.

Formalising some common annotations across various artifact types will make them useful for registries to use to better understand the artifacts.

The future

There is a clear trend towards more types of artifacts being stored in container registries. While it's too early to predict exactly what will be stored in registries, a couple of observations can be made.

Just as how OCI indices evolved from a need, it's likely that the OCI will specify new schemas and media types once there are several new artifacts being stored in registries. This needs to be done, in part, with hindsight so that the specification captures the requirements for storing all artifacts.

It's likely that other artifacts will also want to reference multiple images and/or artifacts. This means that the approach of using an OCI index for a single object is likely to continue to be valid and useful.

Agreeing upon and using several common annotations will provide a solid foundation for future specification work. If Helm, CNAB, and whatever comes next find annotations that work well, these can be promoted to fields of the next OCI specification.

FAQ

What is CNAB?

CNAB stands for Cloud Native Application Bundle. It aims to be the equivalent of a deb (or MSI) package but for all things Cloud Native. See this site for more.

What is an OCI index?

A container image is presented by a registry as a manifest. Each manifest is platform specific which means that in order to use an image on multiple platforms, one needs to fetch the correct manifest for that platform.

Initially this was solved by indicating the platform as part of the tag, e.g.: myimage:tag-<platform>. This is undesirable for base images used on multiple platforms as it requires platform specific code. As such a manifest list was added where multiple manifests could be presented behind the same code.

The client can fetch the manifest list (or OCI index) and match its platform to those presented so that it gets the correct image manifest. Registries are content addressable so the manifest can be found using the digest.

An example of this is the golang:alpine image, note that a Docker manifest list is the older version of an OCI index and they serve the same purpose:

$ DOCKER_CLI_EXPERIMENTAL=enabled docker manifest inspect golang:alpine
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1365,
         "digest": "sha256:9ba4afd1011b9151c3967651538b600f19e48eff2ddde987feb2b72ab2c0bb69",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1572,
         "digest": "sha256:89cc8193f7abc4237b0df2417e0b9fa61687017cd507456b21241d9ea4d94dd3",
         "platform": {
            "architecture": "arm",
            "os": "linux",
            "variant": "v6"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1572,
         "digest": "sha256:6803d818bb3dd6edfccbe35b70477483fc75ed11d925e80e4af443f737146328",
         "platform": {
            "architecture": "arm64",
            "os": "linux",
            "variant": "v8"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1572,
         "digest": "sha256:923201b72b9dcf9e96290f9f171c34a9c743047d707afe69d9c167d430607db7",
         "platform": {
            "architecture": "386",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1572,
         "digest": "sha256:1782df1f6fa4ede250547f7aa491d3d11fa974bc62a1b9b0e493f07c3ba4430f",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 1572,
         "digest": "sha256:d50b32798b5e99eb046ee557c567c83d25e39cbaf42f1d4f24af708644a69123",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Getting started

Clone the project into your GOPATH. You can then build it using:

$ make

You should now have a cnab-to-oci binary the bin/ folder. To run it, execute:

$ bin/cnab-to-oci --help

Prerequisites

  • Make
  • Golang 1.9+
  • Git

Installing

For installing, make sure your $GOPATH/bin is part of your $PATH.

$ make install

This will build and install cnab-to-oci into $GOPATH/bin.

Usage

The cnab-to-oci binary is a demonstration tool to push and pull a CNAB to a registry. It has three commands: push, pull and fixup which are described in the following sections.

Push

The push command packages a bundle.json file into an OCI image index (falling back to a Docker manifest if the registry does not support this) and pushes this to the registry. As part of this process, fixup process is implicitly run.

$ bin/cnab-to-oci push examples/helloworld-cnab/bundle.json --target myhubusername/repo
Ensuring image cnab/helloworld:0.1.1 is present in repository docker.io/myhubusername/repo
Image is not present in repository
Mounting descriptor sha256:bbffe37bb3899b1384bf1483cdcff44bd148d52078b4655e69cd23d534ea043d with media type application/vnd.docker.image.rootfs.diff.tar.gzip (size: 203)
Mounting descriptor sha256:e280b57a032b8bb2ab45f26ea67f42b5d47fd5aca2dd63c5bcdbbd1f753f20b7 with media type application/vnd.docker.image.rootfs.diff.tar.gzip (size: 370)
Mounting descriptor sha256:8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6 with media type application/vnd.docker.image.rootfs.diff.tar.gzip (size: 2206542)
Mounting descriptor sha256:58e6f39290459b6563b348052b2a1a8cf2a44fac19a80ae0da36c82a32f151f8 with media type application/vnd.docker.container.image.v1+json (size: 2135)
Copying descriptor sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6 with media type application/vnd.docker.distribution.manifest.v2+json (size: 942)
{"errors":[{"code":"MANIFEST_INVALID","message":"manifest invalid","detail":{}}]}

Pushed successfully, with digest "sha256:6cabd752cb01d2efb9485225baf7fc26f4322c1f45f537f76c5eeb67ba8d83e0"

Note: if your images -invocation images as well as service images- are not already pushed on a registry, cnab-to-oci will try to resolve them locally and push them from your docker daemon image store.

Note: The MANIFEST_INVALID error in the above case is because the Docker Hub does not currently support the OCI image index type.

Note: When using the Docker Hub, no tag will show up in the Hub interface. The artifact must be referenced by its SHA - see pull.

Pull

The pull command is used to fetch a CNAB packaged as an OCI image index or Docker manifest from a registry. This must be done using the digest returned by the push command. By default the output is saved to pulled.json.

$ bin/cnab-to-oci pull myhubusername/repo@sha256:6cabd752cb01d2efb9485225baf7fc26f4322c1f45f537f76c5eeb67ba8d83e0

$ cat pulled.json
{
 "name": "helloworld",
 "version": "0.1.1",
 "description": "A short description of your bundle",
 "keywords": [
  "helloworld",
  "cnab",
  "tutorial"
 ],
 "maintainers": [
  {
   "name": "Jane Doe",
   "email": "[email protected]",
   "url": "https://example.com"
  }
 ],
 "invocationImages": [
  {
   "imageType": "docker",
   "image": "myhubusername/repo@sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6",
   "size": 942,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json"
  }
 ],
 "images": null,
 "parameters": null,
 "credentials": null
}

Fixup

The fixup command resolves all the image digest references (for the invocationImages as well as the images in the bundle.json) from the relevant registries and pushes them to the target repository to ensure they're available to anyone who has access to the CNAB in the target repository. A patched bundle.json is saved by default to fixed-bundle.json

$ bin/cnab-to-oci fixup examples/helloworld-cnab/bundle.json --target myhubusername/repo
Ensuring image cnab/helloworld:0.1.1 is present in repository docker.io/myhubusername/repo
Image is not present in repository
Mounting descriptor sha256:bbffe37bb3899b1384bf1483cdcff44bd148d52078b4655e69cd23d534ea043d with media type application/vnd.docker.image.rootfs.diff.tar.gzip (size: 203)
Mounting descriptor sha256:e280b57a032b8bb2ab45f26ea67f42b5d47fd5aca2dd63c5bcdbbd1f753f20b7 with media type application/vnd.docker.image.rootfs.diff.tar.gzip (size: 370)
Mounting descriptor sha256:8e3ba11ec2a2b39ab372c60c16b421536e50e5ce64a0bc81765c2e38381bcff6 with media type application/vnd.docker.image.rootfs.diff.tar.gzip (size: 2206542)
Mounting descriptor sha256:58e6f39290459b6563b348052b2a1a8cf2a44fac19a80ae0da36c82a32f151f8 with media type application/vnd.docker.container.image.v1+json (size: 2135)
Copying descriptor sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6 with media type application/vnd.docker.distribution.manifest.v2+json (size: 942)

$ cat fixed-bundle.json
{
  "name": "helloworld",
  "version": "0.1.1",
  "description": "A short description of your bundle",
  "keywords": [
    "helloworld",
    "cnab",
    "tutorial"
  ],
  "maintainers": [
    {
      "name": "Jane Doe",
      "email": "[email protected]",
      "url": "https://example.com"
    }
  ],
  "invocationImages": [
    {
      "imageType": "docker",
      "image": "myhubusername/repo@sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6",
      "size": 942,
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json"
    }
  ],
  "images": null,
  "parameters": null,
  "credentials": null
}

Note: In the above example, the invocation image reference now matches the target repository.

Example

The following is an example of an OCI image index sent to the registry.

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:d59a1aa7866258751a261bae525a1842c7ff0662d4f34a355d5f36826abc0341",
      "size": 285,
      "annotations": {
        "io.cnab.manifest.type": "config"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:196d12cf6ab19273823e700516e98eb1910b03b17840f9d5509f03858484d321",
      "size": 506,
      "annotations": {
        "io.cnab.manifest.type": "invocation"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:6bb891430fb6e2d3b4db41fd1f7ece08c5fc769d8f4823ec33c7c7ba99679213",
      "size": 507,
      "annotations": {
        "io.cnab.component.name": "image-1",
        "io.cnab.manifest.type": "component"
      }
    }
  ],
  "annotations": {
    "io.cnab.keywords": "[\"keyword1\",\"keyword2\"]",
    "io.cnab.runtime_version": "v1.0.0",
    "org.opencontainers.artifactType": "application/vnd.cnab.manifest.v1",
    "org.opencontainers.image.authors": "[{\"name\":\"docker\",\"email\":\"[email protected]\",\"url\":\"docker.com\"}]",
    "org.opencontainers.image.description": "description",
    "org.opencontainers.image.title": "my-app",
    "org.opencontainers.image.version": "0.1.0"
  }
}

The first manifest in the manifest list references the CNAB configuration. An example of this follows:

{
  "schemaVersion": 2,
  "config": {
    "mediaType": "application/vnd.cnab.config.v1+json",
    "digest": "sha256:4bc453b53cb3d914b45f4b250294236adba2c0e09ff6f03793949e7e39fd4cc1",
    "size": 578
  },
  "layers": []
}

Subsequent manifests in the manifest list are standard OCI images.

This example proposes two OCI specification and registry changes:

  1. It proposes the addition of an org.opencontainers.artifactType annotation to be included in the OCI specification.
  2. It requires that registries support the application/vnd.cnab.config.v1+json media type for a config type.

Development

Running the tests

$ make test

Running the e2e tests

$ make e2e

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Maintainers

See also the list of maintainers who participated in this project.

Contributors

See also the list of contributors who participated in this project.

License

This project is licensed under the Apache 2 License - see the LICENSE file for details.

cnab-to-oci's People

Contributors

carolynvs avatar carolynvs-msft avatar chris-crone avatar dependabot[bot] avatar eunomie avatar jcsirot avatar jeremyrickard avatar ndeloof avatar radu-matei avatar rumpl avatar silvin-lubecki avatar simonferquel avatar trishankatdatadog avatar ulyssessouza avatar vdice avatar vinozzz avatar zappy-shu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cnab-to-oci's Issues

Switch canonical json libraries

We switched canonical json libraries in the CNAB spec and cnab-go, but missed updating this repo at the same time which prevents pushing bundles that use numeric json types (e.g. the value 0.5 cannot be used in a bundle).

We should switch to using github.com/cyberphone/json-canonicalization and stop using the docker canonical json serializer.

Decide enforcing tags for bundles

The current behaviour of cnab-to-oci is to allow pushing to a repository without a tag:

$ cnab-to-oci push testdata/cnab/bundle.json --target localhost:5000/no-tag-test
Starting to copy image cnab/helloworld:0.1.1...
Completed image cnab/helloworld:0.1.1 copy
Pushed successfully, with digest "sha256:b4936e42304c184bafc9b06dde9ea1f979129e09a021a8f40abc07f736de9268"

$ cnab-to-oci pull localhost:5000/no-tag-test                                   
Error: failed to resolve bundle manifest "localhost:5000/no-tag-test": object required

$ cnab-to-oci pull localhost:5000/no-tag-test@sha256:b4936e42304c184bafc9b06dde9ea1f979129e09a021a8f40abc07f736de9268

If a bundle is pushed without a tag, pulling it has to be performed using the full SHA256 digest, which is unknown to a potential consumer of the bundle.
Should we enforce pushing with a specific tag? (Or at least by default?)

Note that this is also significant in the context of signing.

Release cnab-to-oci

To prepare next CNAB 1.0 release, we need to release a first version of cnab-to-oci so projects depending on it can pin this version.

Configure log level

As a tool to discover and learn how push/pull works, I think we should have a real logging policy, so a user can check what is pushed, when and where.

We should add some debug logging, and a way to configure log level.

Configurable annotations for for the index

Looking at the raw index that this library generates, the first question I have is whether we can have support for configurable annotations - specifically, I'm looking at:

"io.docker.app.format": "cnab",
"io.docker.type": "app",

Does Docker App have a strong dependency on these two annotations?

Thanks!

Copy a bundle with a relocation map to another registry

Here's the scenario that a Porter user is encountering:

  1. Push a bundle to a public registry, example/mybun:v1.0.0
  2. Copy the bundle to local registry, localhost:5000/mybun:v1.0.0. The copy now has a relocation map that maps example/mybun-invocation-image:v1.0.0 => localhost:5000/mybun-invocation-image:v1.0.0.
  3. Disconnect your internet connection.
  4. Copy the bundle from localhost:5000/mybun:v1.0.0 to localhost:5000/newbun:v1.0.0.

The copy fails because it tries to copy the invocation image from its original location defined in the bundle, instead of copying the relocated invocation image.

I'm not sure if there is a way to make this work right now? It seems like FixupBundle should take a relocation map for the bundle, and take any relocations into account before pulling and copying images?

Pushing a CNAB bundle to localhost TLS Registry fails

Steps to reproduce:

  1. Run scripts/notary-start.sh.
 $ signy --tlscacert=/Users/trishank.kuppusamy/go/src/github.com/theupdateframework/notary/cmd/notary/root-ca.crt --server=https://localhost:4443 --log=debug sign testdata/cnab/bundle.json localhost:5000/cnab/helloworld:0.1.1
DEBU[0000] Fixing up bundle localhost:5000/cnab/helloworld:0.1.1
DEBU[0000] Updating entry in relocation map for "cnab/helloworld:0.1.1"
INFO[0000] Starting to copy image cnab/helloworld:0.1.1
INFO[0000] Failed to copy image cnab/helloworld:0.1.1: failed to do request: Head http://localhost:5000/v2/cnab/helloworld/blobs/sha256:58e6f39290459b6563b348052b2a1a8cf2a44fac19a80ae0da36c82a32f151f8: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
Error: failed to do request: Head http://localhost:5000/v2/cnab/helloworld/blobs/sha256:58e6f39290459b6563b348052b2a1a8cf2a44fac19a80ae0da36c82a32f151f8: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
Usage:
  signy sign [file] [target reference] [flags]

Flags:
  -h, --help                help for sign
      --in-toto             Adds in-toto metadata to TUF. If passed, the root layout, links directory, and root kyes must be supplied
      --layout string       Path to the in-toto root layout file
      --layout-key string   Path to the in-toto root layout public keys
      --links string        Path to the in-toto links directory
      --root-key string     Root key to initialize the repository with
      --thick               Signs a thick bundle. If passed, only the signature is pushed to the trust server, not the bundle file

Global Flags:
  -d, --dir string         Directory where the trust data is persisted to (default "/Users/trishank.kuppusamy/.signy")
      --log string         Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info")
      --server string      The trust server used (default "https://notary.docker.io")
  -t, --timeout string     Timeout for the trust server (default "5s")
      --tlscacert string   Trust certs signed only by this CA

failed to do request: Head http://localhost:5000/v2/cnab/helloworld/blobs/sha256:58e6f39290459b6563b348052b2a1a8cf2a44fac19a80ae0da36c82a32f151f8: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
  1. Run scripts/stop.sh.

  2. Run scripts/signy-start.sh.

signy --tlscacert=/Users/trishank.kuppusamy/go/src/github.com/theupdateframework/notary/cmd/notary/root-ca.crt --server=https://localhost:4443 --log=debug sign testdata/cnab/bundle.json localhost:5000/cnab/helloworld:0.1.1
DEBU[0000] Fixing up bundle localhost:5000/cnab/helloworld:0.1.1
DEBU[0000] Updating entry in relocation map for "cnab/helloworld:0.1.1"
INFO[0000] Starting to copy image cnab/helloworld:0.1.1
INFO[0001] Completed image cnab/helloworld:0.1.1 copy
DEBU[0001] Bundle fixed
INFO[0001] Generated relocation map: relocation.ImageRelocationMap{"cnab/helloworld:0.1.1":"localhost:5000/cnab/helloworld@sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6"}
DEBU[0001] Pushing CNAB Bundle localhost:5000/cnab/helloworld:0.1.1
DEBU[0001] Pushing CNAB Bundle Config
DEBU[0001] Trying to push CNAB Bundle Config
DEBU[0001] CNAB Bundle Config Descriptor
DEBU[0001] {
  "mediaType": "application/vnd.cnab.config.v1+json",
  "digest": "sha256:c7e92bd51f059d60b15ad456edf194648997d739f60799b37e08edafd88a81b5",
  "size": 501
}
DEBU[0001] Trying to push CNAB Bundle Config Manifest
DEBU[0001] CNAB Bundle Config Manifest Descriptor
DEBU[0001] {
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "digest": "sha256:c88087935c91817e3421c41794ace533f597428d4a9617bf7a6de5bc4200d8da",
  "size": 188
}
DEBU[0001] CNAB Bundle Config pushed
DEBU[0001] Pushing CNAB Index
DEBU[0001] Trying to push OCI Index
DEBU[0001] {"schemaVersion":2,"manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:c88087935c91817e3421c41794ace533f597428d4a9617bf7a6de5bc4200d8da","size":188,"annotations":{"io.cnab.manifest.type":"config"}},{"mediaType":"application/vnd.docker.distribution.manifest.v2+json","digest":"sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6","size":942,"annotations":{"io.cnab.manifest.type":"invocation"}}],"annotations":{"io.cnab.keywords":"[\"helloworld\",\"cnab\",\"tutorial\"]","io.cnab.runtime_version":"v1.0.0-WD","org.opencontainers.artifactType":"application/vnd.cnab.manifest.v1","org.opencontainers.image.authors":"[{\"name\":\"Jane Doe\",\"email\":\"[email protected]\",\"url\":\"https://example.com\"}]","org.opencontainers.image.description":"A short description of your bundle","org.opencontainers.image.title":"helloworld","org.opencontainers.image.version":"0.1.1"}}
DEBU[0001] OCI Index Descriptor
DEBU[0001] {
  "mediaType": "application/vnd.oci.image.index.v1+json",
  "digest": "sha256:b4936e42304c184bafc9b06dde9ea1f979129e09a021a8f40abc07f736de9268",
  "size": 929
}
DEBU[0001] CNAB Index pushed
DEBU[0001] CNAB Bundle pushed
INFO[0001] Pushed successfully, with digest "sha256:b4936e42304c184bafc9b06dde9ea1f979129e09a021a8f40abc07f736de9268"
DEBU[0001] cannot get default credentials: authentication not found for trust server https://localhost:4443
DEBU[0001] Making dir path: /Users/trishank.kuppusamy/.signy/tuf/localhost/changelist
DEBU[0001] entered ValidateRoot with dns: localhost
DEBU[0001] found the following root keys: [f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9]
DEBU[0001] found 1 valid leaf certificates for localhost: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] found 1 leaf certs, of which 1 are valid leaf certs for localhost
DEBU[0001] checking root against trust_pinning config for localhost
DEBU[0001] checking trust-pinning for cert: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001]  role has key IDs: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] verifying signature for key ID: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] root validation succeeded for localhost
DEBU[0001] entered ValidateRoot with dns: localhost
DEBU[0001] found the following root keys: [f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9]
DEBU[0001] found 1 valid leaf certificates for localhost: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] found 1 leaf certs, of which 1 are valid leaf certs for localhost
DEBU[0001] checking root against trust_pinning config for localhost
DEBU[0001] checking trust-pinning for cert: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001]  role has key IDs: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] verifying signature for key ID: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] root validation succeeded for localhost
DEBU[0001] updating TUF client
DEBU[0001] Loading timestamp...
DEBU[0001] 200 when retrieving metadata for timestamp
DEBU[0001] timestamp role has key IDs: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] verifying signature for key ID: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] timestamp role has key IDs: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] verifying signature for key ID: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] successfully verified downloaded timestamp
DEBU[0001] Loading snapshot...
DEBU[0001] cached snapshot is invalid (must download): sha256 checksum for snapshot did not match: expected bf3b30295e102d65d567c2644980d748ebe8b1c8b1981c46edbabd547ac75512
DEBU[0001] 200 when retrieving metadata for snapshot.bf3b30295e102d65d567c2644980d748ebe8b1c8b1981c46edbabd547ac75512
DEBU[0001] snapshot role has key IDs: 83abf5bc3119245b26d6af7542f87a8c30e625d3cc62078123d517f2ad48fc80
DEBU[0001] verifying signature for key ID: 83abf5bc3119245b26d6af7542f87a8c30e625d3cc62078123d517f2ad48fc80
DEBU[0001] snapshot role has key IDs: 83abf5bc3119245b26d6af7542f87a8c30e625d3cc62078123d517f2ad48fc80
DEBU[0001] verifying signature for key ID: 83abf5bc3119245b26d6af7542f87a8c30e625d3cc62078123d517f2ad48fc80
DEBU[0001] successfully verified downloaded snapshot.bf3b30295e102d65d567c2644980d748ebe8b1c8b1981c46edbabd547ac75512
DEBU[0001] Loading targets...
DEBU[0001] targets role has key IDs: c3bfdf9b15f43aebe73ae2011c3b101176c448a69057e48867a2cfab0ec30c97
DEBU[0001] verifying signature for key ID: c3bfdf9b15f43aebe73ae2011c3b101176c448a69057e48867a2cfab0ec30c97
DEBU[0001] successfully verified cached targets
DEBU[0001] Adding target "5000/cnab/helloworld" with sha256 "c7e92bd51f059d60b15ad456edf194648997d739f60799b37e08edafd88a81b5" and size 501 bytes.
DEBU[0001] entered ValidateRoot with dns: localhost
DEBU[0001] found the following root keys: [f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9]
DEBU[0001] found 1 valid leaf certificates for localhost: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] found 1 leaf certs, of which 1 are valid leaf certs for localhost
DEBU[0001] checking root against trust_pinning config for localhost
DEBU[0001] checking trust-pinning for cert: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001]  role has key IDs: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] verifying signature for key ID: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] root validation succeeded for localhost
DEBU[0001] entered ValidateRoot with dns: localhost
DEBU[0001] found the following root keys: [f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9]
DEBU[0001] found 1 valid leaf certificates for localhost: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] found 1 leaf certs, of which 1 are valid leaf certs for localhost
DEBU[0001] checking root against trust_pinning config for localhost
DEBU[0001] checking trust-pinning for cert: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001]  role has key IDs: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] verifying signature for key ID: f01c4109378763e9908eeed725c691586aa7c1b735c312989f64270f7925a9b9
DEBU[0001] root validation succeeded for localhost
DEBU[0001] 200 when retrieving metadata for root
DEBU[0001] updating TUF client
DEBU[0001] Loading timestamp...
DEBU[0001] 200 when retrieving metadata for timestamp
DEBU[0001] timestamp role has key IDs: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] verifying signature for key ID: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] timestamp role has key IDs: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] verifying signature for key ID: 919e5d9116881bfdfb2cc8d02f4836b2da7894c6c4bb65a0078333228aff945d
DEBU[0001] successfully verified downloaded timestamp
DEBU[0001] Loading snapshot...
DEBU[0001] snapshot role has key IDs: 83abf5bc3119245b26d6af7542f87a8c30e625d3cc62078123d517f2ad48fc80
DEBU[0001] verifying signature for key ID: 83abf5bc3119245b26d6af7542f87a8c30e625d3cc62078123d517f2ad48fc80
DEBU[0001] successfully verified cached snapshot
DEBU[0001] Loading targets...
DEBU[0001] targets role has key IDs: c3bfdf9b15f43aebe73ae2011c3b101176c448a69057e48867a2cfab0ec30c97
DEBU[0001] verifying signature for key ID: c3bfdf9b15f43aebe73ae2011c3b101176c448a69057e48867a2cfab0ec30c97
DEBU[0001] successfully verified cached targets
DEBU[0001] changelist add: 5000/cnab/helloworld
Enter passphrase for targets key with ID c3bfdf9:
DEBU[0005] applied 1 change(s)
DEBU[0005] signing snapshot...
DEBU[0005] sign called with 1/1 required keys
Enter passphrase for snapshot key with ID 83abf5b:
DEBU[0006] sign called with 0/0 required keys
INFO[0006] Pushed trust data for localhost:5000/cnab/helloworld:0.1.1: c7e92bd51f059d60b15ad456edf194648997d739f60799b37e08edafd88a81b5

Most likely has to do with using self-signed TLS cert.

Push a CNAB Bundle to GCR fails

Pushing a CNAB bundle to gcr.io returns an error:

Error: error while pushing bundle config manifest: unexpected status: 400 Bad Request

After some investigation, we found multiple caveats:

  • The bundle manifest itself had only a config field and a null layer field. This was fixed in #64 , adding a dummy layer pointing to the config object itself (it adds duplication)
  • When using external (mostly multi architecture images) images for services, we are potentially mixing different format representations: one service can be a manifest list, another an OCI index, and the CNAB itself be an OCI index too. This mix is not supported by gcr (and maybe other registries too). I think we can't guarantee that every service image will share the same format, so the minimum we can do is to fail if we detect a mix between images. That said, we need to also improve cnab-to-oci to detect which format we should use, depending the one found in the service images, instead of the fallback mechanism (try with OCI Index then Docker Manifest List).

The media type used doesn't meet the OCI artifact spec

The media type of CNAB config artifact is application/vnd.cnab.config.v1+json which doesn't meet the OCI artifact spec standard for the unique type: application/vnd.[org|company].[objectType].[optionalSubType].config.[version]+json

How about add a cncf section: application/vnd.cncf.cnab.config.v1+json?

Image copy fails when image contains duplicate layers

Some images have duplicate layers, for example docker/whalesay:

$ docker image inspect docker/whalesay
...
"RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:1154ba695078d29ea6c4e1adb55c463959cd77509adf09710e2315827d66271a",
                "sha256:528c8710fd95f61d40b8bb8a549fa8dfa737d9b9c7c7b2ae55f745c972dddacd",
                "sha256:37ee47034d9b78f10f0c5ce3a25e6b6e58997fcadaf5f896c603a10c5f35fb31",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:b26122d57afa5c4a2dc8db3f986410805bc8792af3a4fa73cfde5eed0a8e5b6d",
                "sha256:091abc5148e4d32cecb5522067509d7ffc1e8ac272ff75d2775138639a6c50ca",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef",
                "sha256:d511ed9e12e17ab4bfc3e80ed7ce86d4aac82769b42f42b753a338ed9b8a566d",
                "sha256:d061ee1340ecc8d03ca25e6ca7f7502275f558764c1ab46bd1f37854c74c5b3f",
                "sha256:5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef"
            ]
        },

Note that 5f70bf18a086007016e948b04aed3b82103a36bea41755b6cddfaf10ace3c6ef is in there multiple times. This can happen when a RUN command doesn't modify the file system, such as running COPY + RUN chmod vs COPY --chmod.

We are simultaneously kicking off copies for each layer, and if there are duplicates, then a race begins over which copy will run successfully and which will notice that it didn't write the expected number of bytes (because the layer already exists).

This causes the following error in Porter (see getporter/porter#2320):

error preparing the bundle with cnab-to-oci before pushing: failed commit on ref "layer-sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4": unexpected size 64, expected 32.

We should only copy the unique set of layers in an image, and not kick off extra copies for duplicate layers.

Handle bundle custome extensions

The cnab spec provides a mechanism for bundle authors to define custom extensions. It looks like cnab-to-oci currently does not handle these when a bundle is pushed, so this data would be lost on a pull.

Porter plans on storing some things in the extensions and would like to see this persisted.

Surface error(s) from fixup to API consumers

This is a question-request around errors that may result from various image resolving/pushing fixups as performed in remotes/fixup.go, specifically centering around this line: https://github.com/cnabio/cnab-to-oci/blob/master/remotes/fixup.go#L217

Currently the error appears to be logged in a debug fashion. Is there a way for API consumers to tap into the debug logging in this case? Or, would it make sense to immediately return the error if one occurs?

Perhaps, if a given fixup error is acceptable and the rest of the fixup functions are intended to be called, I wonder if we update the code to track/append any/all fixup errors and include this in the final returned error upon exiting the function? (Note how no errors are wrapped in https://github.com/cnabio/cnab-to-oci/blob/master/remotes/fixup.go#L224)

A recent issue spurring these thoughts cropped up in getporter/porter#1032. Here, porter utilizes this library to publish a bundle and invocation images, but currently it doesn't have visibility into any fixup errors that might occur.

Store the whole Bundle.json and generate a Relocation map

Context

During CNAB discussions about storing bundles into registry, some comments were raised about cnab-to-oci not storing the whole bundle.json in the registry but only a subset, reconstructing it during a pull.

Discussion around storing the complete bundle.json file (and duplicate the information about the reference images) vs. reconstructing it based on a configuration file (parameters, credentials, metadata - all other fields in bundle.json) and the referenced images. Whatever path we choose, the proposed changes to the OCI specification should allow both options. Same goes for signing - whether we choose to sign the complete bundle.json that is stored in the registry, or we sign the manifest generated by the registry, the proposed changes to the specification and the current CNAB security spec should allow for for both approaches.

Current behavior

When pushing a CNAB Bundle to a registry, cnab-to-oci produces the following:

  • Push or Mount all the invocation and service images inside the same repository, and retrieve their digested reference
  • Create a new config object, storing only a sub-set of the bundle.json (see the type here (with mostly parameters, actions and credentials) and retrieve its digest
    Ex:
{
  "schemaVersion": "v1.0.0-WD",
  "actions": {
    "com.docker.app.inspect": {
      "stateless": true
    },
    "com.docker.app.render": {
      "stateless": true
    },
    "io.cnab.status": {}
  },
  "definitions": {
    "port": {
      "default": "8080",
      "type": "string"
    },
    "text": {
      "default": "Hello, World!",
      "type": "string"
    }
  },
  "parameters": {
    "fields": {
      "port": {
        "definition": "port",
        "destination": {
          "env": "docker_param1"
        }
      },
      "text": {
        "definition": "text",
        "destination": {
          "env": "docker_param2"
        }
      }
    }
  },
  "credentials": {
    "docker.context": {
      "path": "/cnab/app/context.dockercontext"
    }
  }
} 
  • Create an OCI Index (or Manifest List) with some informations found in the bundle.json like metadata (as annotations), or invocation/service images
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:1f1470cc6d80bcf8057782bc38a2469a3d4c2833e804b98c19b6f4dedcddda4e",
      "size": 316,
      "annotations": {
        "io.cnab.manifest.type": "config"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:720d8fa437af27e98a76ade5395b92c32a8e39721c62a6b4d8c3e4e1967d0caa",
      "size": 1363,
      "annotations": {
        "io.cnab.manifest.type": "invocation"
      }
    },
    {
      "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
      "digest": "sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96",
      "size": 528,
      "annotations": {
        "io.cnab.component.name": "hello",
        "io.cnab.manifest.type": "component"
      }
    }
  ],
  "annotations": {
    "io.cnab.runtime_version": "v1.0.0-WD",
    "io.docker.app.format": "cnab",
    "io.docker.type": "app",
    "org.opencontainers.artifactType": "application/vnd.cnab.manifest.v1",
    "org.opencontainers.image.authors": "[{\"name\":\"user\",\"email\":\"[email protected]\"}]",
    "org.opencontainers.image.description": "Hello, World!",
    "org.opencontainers.image.title": "hello-world",
    "org.opencontainers.image.version": "0.1.0"
  }
} 

During the pull, we can reconstruct the bundle.json from the config object, the annotations and the images:

{
        "schemaVersion": "v1.0.0-WD",
        "name": "hello-world",
        "version": "0.1.0",
        "description": "Hello, World!",
        "maintainers": [
                {
                        "name": "user",
                        "email": "[email protected]"
                }
        ],
        "invocationImages": [
                {
                        "imageType": "docker",
                        "image": "slubecki/test1@sha256:720d8fa437af27e98a76ade5395b92c32a8e39721c62a6b4d8c3e4e1967d0caa",
                        "size": 1363,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json"
                }
        ],
        "images": {
                "hello": {
                        "imageType": "docker",
                        "image": "slubecki/test1@sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96",
                        "size": 528,
                        "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
                        "description": ""
                }
        },
        "actions": {
                "com.docker.app.inspect": {
                        "stateless": true
                },
                "com.docker.app.render": {
                        "stateless": true
                },
                "io.cnab.status": {}
        },
        "parameters": {
                "fields": {
                        "port": {
                                "definition": "port",
                                "destination": {
                                        "env": "docker_param1"
                                }
                        },
                        "text": {
                                "definition": "text",
                                "destination": {
                                        "env": "docker_param2"
                                }
                        }
                }
        },
        "credentials": {
                "docker.context": {
                        "path": "/cnab/app/context.dockercontext"
                }
        },
        "definitions": {
                "port": {
                        "default": "8080",
                        "type": "string"
                },
                "text": {
                        "default": "Hello, World!",
                        "type": "string"
                }
        }
}

This reconstruction was designed to tackle 2 things:

  • avoiding duplication between information stored at OCI Index level (which can be directly parsed by registries)
  • moving a CNAB from a repository to another means we need to update the bundle.json image references, and then to re-sign it, so moving would remove any guarantee seal. With this design moving an OCI index does not change anything, we can safely keep the signature.

Proposal

Since this design was implemented, the Image Relocation has been added to the CNAB specifications.

We propose to leverage this new feature in the Bundle Runtime so:

  • whole bundle can be pushed as is, as relocating it does not need to patch the bundle but just generates locally a relocation map. cnab-to-oci can then pull the original bundle.json.
  • cnab-to-oci generates the relocation map from the image references in the OCI Index.

Ex of Relocation map:

{
 "hashicorp/http-echo": "slubecki/test1@sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96"
}

This new design does not avoid annotation/metadata replication, but we think it is a minor issue, regarding the concern of storing a subset of the bundle.json.

Test Flake: TestPromisesScheduleErroredDontBlockDontRun

I have noticed that TestPromisesScheduleErroredDontBlockDontRun fails occasionally and then passes when I retrigger the build. I never get it to fail locally though.

CGO_ENABLED=0 go build -ldflags="-s -w -X github.com/cnabio/cnab-to-oci/internal.GitCommit=8681d4f -X github.com/cnabio/cnab-to-oci/internal.Version=v0.3.1-beta1-10-g8681d4f -X github.com/cnabio/cnab-to-oci/internal.BuildTime=2022-03-15T14:32:58Z" -o bin/cnab-to-oci ./cmd/cnab-to-oci
go test -ldflags="-s -w -X github.com/cnabio/cnab-to-oci/internal.GitCommit=8681d4f -X github.com/cnabio/cnab-to-oci/internal.Version=v0.3.1-beta1-10-g8681d4f -X github.com/cnabio/cnab-to-oci/internal.BuildTime=2022-03-15T14:32:58Z" -failfast -race github.com/cnabio/cnab-to-oci github.com/cnabio/cnab-to-oci/cmd/cnab-to-oci github.com/cnabio/cnab-to-oci/converter github.com/cnabio/cnab-to-oci/internal github.com/cnabio/cnab-to-oci/relocation github.com/cnabio/cnab-to-oci/remotes github.com/cnabio/cnab-to-oci/tests
?   	github.com/cnabio/cnab-to-oci	[no test files]
?   	github.com/cnabio/cnab-to-oci/cmd/cnab-to-oci	[no test files]
ok  	github.com/cnabio/cnab-to-oci/converter	0.193s
?   	github.com/cnabio/cnab-to-oci/internal	[no test files]
?   	github.com/cnabio/cnab-to-oci/relocation	[no test files]
time="2022-03-15T14:34:59Z" level=warning msg="reference for unknown type: "
--- FAIL: TestPromisesScheduleErroredDontBlockDontRun (0.00s)
    promises_test.go:87: assertion failed: expected an error, got nil
FAIL
FAIL	github.com/cnabio/cnab-to-oci/remotes	0.124s
?   	github.com/cnabio/cnab-to-oci/tests	[no test files]
FAIL
make: *** [Makefile:68: test-unit] Error 1

Exited with code exit status 2

Cannot build on Ubuntu 20.04

The make target is looking for gcc-5 which is really old and not available on Ubuntu 20.04. I saw a PR #47 to update it but that has been open for almost a year. Are there any plans to update the build tooling so this tool can continue to be used?

Release v0.3.0-beta1

After we merge #78, I propose we do a new release so projects that use cnab-to-oci can cleanly use the newly added features.

Is there anything else we would want in this release?
I'd very much propose we cut the release sooner rather than later, and just release again if we have other features.

Pushing the same bundle twice yields different digest

Pushing the same bundle twice without modification results in a different digest.

I suspect this is because of the image map which is unordered. To fix this, we’ll need to:

  • use an ordered map (or list) when serialising
  • document how the above works (likely in the CNAB spec) so that others can implement it too

Perform TUF metadata pushing and verification

Should this library be responsible for:

  • pushing the metadata when pushing a bundle?
  • pulling and verifying the metadata when pulling a bundle?

Or should it be the responsibility of the tooling that uses the library?

Thoughts?

Forked containerd/containerd dependency

This library is using a forked version of containerd/containerd, and while upstream is at version 1.26, this is at 1.21, and I have a couple of questions:

  • is there a plan for the changes in the fork to be pushed upstream?
  • if that is taking longer, is there a plan to keep the branch used here up-to-date?

Thanks!

Push images if not present in registry

I've been going back and forth over this for the last couple of weeks - consider the following scenario - a bundle was just built locally, and the images have not been pushed - before calling cnab-to-oci to push the bundle, runtimes must first push the images to a repository. Then, cnab-to-oci copies those images to a new repository, which is rather inefficient.

Do people have strong opinions against pushing images? I guess a sane workflow would be to checking if the image exists remotely and trying to copy (current behaviour), then fall back to tagging it accordingly and pushing it to the target repository.

Thoughts?

Failed to push the bundle following the steps in readme

root@kube-worker:~/workspace/src/github.com/docker/cnab-to-oci# bin/cnab-to-oci push examples/helloworld-cnab/bundle.json --target myhubusername/repo
Starting to copy image cnab/helloworld:0.1.1...
Error: image "cnab/helloworld:0.1.1" digest differs "" after fixup: "sha256:a59a4e74d9cc89e4e75dfb2cc7ea5c108e4236ba6231b53081a9e2506d1197b6"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.