manifest not found

[hmtai]

manifest not found

[ashank07]

@estesp Apologies, I must have missed the notification from your first ask.

I built the manifest-tool binary from the branch that PR is based on:

./manifest-tool version 1.0.2 (commit: b4e21e57fccdffe1c2929f896a760022be746634)

Looks like that is working, after using that version to push a manifest-list to Artifactory the structure looks correct in the Artifactory UI, and I am able to successfully inspect the manifest using the docker manifest inspect command (which was not working previously after pushing with the 1.0.0 tool):

docker manifest inspect hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank:test
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 527,
         "digest": "sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5",
         "platform": {
            "architecture": "s390x",
            "os": "linux"
         }
      }
   ]
}

Let me know if you need anything else from us.

[estesp]

That's pretty confusing as the last line of your output shows a digest that is a response from the registry side; it also means the registry responded with a successful HTTP response code. Can you try this command?

$ manifest-tool inspect hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/default-http-backend@sha256:77a6a492b57272e8818cd3a300cc40d24c1d87c20275940e7a76101c806d7315

This will ask the registry for the specific digest that it responded with and manifest-tool will dump out details of that media type and contents. If that doesn't respond with anything, then something is broken on the Artifactory side in that it responded with success but didn't actually register the new image manifest, which would be bad.

At that point we would have to investigate server-side logs or use my under development version of manifest tool which dumps HTTP traffic headers/responses.

[hmtai]

@estesp Thanks for your quick response,I try your command:

# manifest-tool inspect hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/default-http-backend@sha256:77a6a492b57272e8818cd3a300cc40d24c1d87c20275940e7a76101c806d7315
FATA[0001] manifest unknown: The named manifest is not known to the registry.

How can I debug this problem further?Could you give me some suggestions?Thanks a lot.

[estesp]

Hmm, this means Artifactory lied to us when it gave us a SHA and a success response code! I'm not sure if you are able to build the v2-rework branch and try with --debug, but that will give all the detail on the HTTP traffic between the registry and manifest-tool given on that branch I switched to using the containerd library for registry interaction. If you can't build that branch let me know and I can provide a test binary so we can debug the Artifactory issue(s).

[ashank07]

Hi Phil, working with some of our developers we noticed that we are seeing different behavior with v0.7.0 and v1.0.0 of the manifest-tool.

I just did a manual test where I downloaded a pause image for 3 different architectures and retagged it just to have something small, giving me these 3 images:

hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x
hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le
hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64

I then ran the following command to upload the manifest list using the v0.7.0 version of the tool and got the following output:

ashankMac:cicd-python-automation ashank$ build-harness/vendor/manifest-tool --debug --username ${ARTIFACTORY_USER} --password ${ARTIFACTORY_TOKEN} push from-args --platforms linux/amd64,linux/ppc64le,linux/s390x --template hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ARCH:test --target hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank:test
DEBU[0000] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc420001b00} {false http://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc420001b00} {false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v1 false true 0xc420001e00} {false http://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v1 false true 0xc420001e00}] 
DEBU[0000] repoName: ibmcom/ashank                      
INFO[0000] Retrieving digests of images...              
DEBU[0000] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc42030e300} {false http://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc42030e300}] 
DEBU[0000] Trying to fetch image manifest of hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64 repository from https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 
INFO[0002] Image "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64:test" is digest sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d; size: 527 
DEBU[0002] Adding manifest references of "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64:test" to blob mount requests 
DEBU[0002] Adding manifest "ibmcom/ashank-amd64" -> to be pushed to "ibmcom/ashank" as a manifest reference 
DEBU[0002] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc42030e480} {false http://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc42030e480}] 
DEBU[0002] Trying to fetch image manifest of hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le repository from https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 
INFO[0012] Image "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le:test" is digest sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0; size: 527 
DEBU[0012] Adding manifest references of "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le:test" to blob mount requests 
DEBU[0012] Adding manifest "ibmcom/ashank-ppc64le" -> to be pushed to "ibmcom/ashank" as a manifest reference 
DEBU[0012] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc4204fc780} {false http://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false true 0xc4204fc780}] 
DEBU[0012] Trying to fetch image manifest of hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x repository from https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 
INFO[0014] Image "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x:test" is digest sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5; size: 527 
DEBU[0014] Adding manifest references of "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x:test" to blob mount requests 
DEBU[0014] Adding manifest "ibmcom/ashank-s390x" -> to be pushed to "ibmcom/ashank" as a manifest reference 
DEBU[0014] Manifest list push url: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test 
DEBU[0014] mediaType of manifestList: application/vnd.docker.distribution.manifest.list.v2+json 
DEBU[0015] Mount of blob sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" 
DEBU[0016] Mount of blob sha256:7675586df687972b960134ddaf042c570c895bf1fbdf9fc0ce0bf09c1e1c2811 succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:7675586df687972b960134ddaf042c570c895bf1fbdf9fc0ce0bf09c1e1c2811" 
DEBU[0016] Mount of blob sha256:1652adb11bd5c0bc4541f6cf812d111180a21bf88dc93ebe626ff223f103ecb8 succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:1652adb11bd5c0bc4541f6cf812d111180a21bf88dc93ebe626ff223f103ecb8" 
DEBU[0017] Mount of blob sha256:4d03cb55d8327070d0de2332365e600ad0b2afd2a30057e29cb3f9fed83e3ea0 succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:4d03cb55d8327070d0de2332365e600ad0b2afd2a30057e29cb3f9fed83e3ea0" 
DEBU[0017] Mount of blob sha256:1a792e8d3c38959ac9672a023ae365b3af7eb3a54016e2ac6a1dd0e782546cec succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:1a792e8d3c38959ac9672a023ae365b3af7eb3a54016e2ac6a1dd0e782546cec" 
DEBU[0020] Mount of blob sha256:f1f9f8769b9c22cdf3fbe980f81cc99a5ed646fb6fab74cbf5ee3f2519a98c2d succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:f1f9f8769b9c22cdf3fbe980f81cc99a5ed646fb6fab74cbf5ee3f2519a98c2d" 
DEBU[0020] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d 
DEBU[0021] referenced manifest "ibmcom/ashank-amd64" pushed; digest matches: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d 
DEBU[0021] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0 
DEBU[0030] referenced manifest "ibmcom/ashank-ppc64le" pushed; digest matches: sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0 
DEBU[0030] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5 
DEBU[0031] referenced manifest "ibmcom/ashank-s390x" pushed; digest matches: sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5 
Digest: sha256:eff4bf0e2ec1d993abc6049495795b01f0ecfbc7c6686dd148f2e1402e7ef450 1051

Looking at the structure this produced on Artifactory via their webui, what we end up with is as follows, which looks correct. The manifest-list is pushed under the test tag, and each of the arch-specific images have been pushed under their own sha (since they are coming from an image repo with a different name):

ashank
  - test
    + list.manifest.json
  - sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5
    + manifest.json
    + sha256__1a792e8d3c38959ac9672a023ae365b3af7eb3a54016e2ac6a1dd0e782546cec
    + sha256__f1f9f8769b9c22cdf3fbe980f81cc99a5ed646fb6fab74cbf5ee3f2519a98c2d
  - sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0
    + manifest.json
    + sha256__1652adb11bd5c0bc4541f6cf812d111180a21bf88dc93ebe626ff223f103ecb8
    + sha256__4d03cb55d8327070d0de2332365e600ad0b2afd2a30057e29cb3f9fed83e3ea0
  - sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
    + manifest.json
    + sha256__7675586df687972b960134ddaf042c570c895bf1fbdf9fc0ce0bf09c1e1c2811
    + sha256__da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e

When I attempt the push using the v1.0.0 version of the manifest-tool, I get the following output:

ashankMac:cicd-python-automation ashank$ manifest-tool --debug --username ${ARTIFACTORY_USER} --password ${ARTIFACTORY_TOKEN} push from-args --platforms linux/amd64,linux/ppc64le,linux/s390x --template hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ARCH:test --target hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank:test
DEBU[0000] hostDir: /etc/docker/certs.d/hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com 
DEBU[0000] hostDir: /etc/docker/certs.d/hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com 
DEBU[0000] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false false true 0xc00006e780} {false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v1 false false true 0xc000001680}] 
DEBU[0000] repoName: ibmcom/ashank                      
INFO[0000] Retrieving digests of images...              
DEBU[0000] hostDir: /etc/docker/certs.d/hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com 
DEBU[0000] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false false true 0xc000001b00}] 
DEBU[0000] Trying to fetch image manifest of hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64 repository from https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 
INFO[0005] Image "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64:test" is digest sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d; size: 527 
DEBU[0005] Adding manifest references of "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-amd64:test" to blob mount requests 
DEBU[0005] Adding manifest "ibmcom/ashank-amd64" -> to be pushed to "ibmcom/ashank" as a manifest reference 
DEBU[0006] hostDir: /etc/docker/certs.d/hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com 
DEBU[0006] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false false true 0xc0002dbe00}] 
DEBU[0006] Trying to fetch image manifest of hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le repository from https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 
INFO[0007] Image "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le:test" is digest sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0; size: 527 
DEBU[0007] Adding manifest references of "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-ppc64le:test" to blob mount requests 
DEBU[0007] Adding manifest "ibmcom/ashank-ppc64le" -> to be pushed to "ibmcom/ashank" as a manifest reference 
DEBU[0008] hostDir: /etc/docker/certs.d/hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com 
DEBU[0008] endpoints: [{false https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 false false true 0xc000543680}] 
DEBU[0008] Trying to fetch image manifest of hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x repository from https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com v2 
INFO[0009] Image "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x:test" is digest sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5; size: 527 
DEBU[0009] Adding manifest references of "hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/ibmcom/ashank-s390x:test" to blob mount requests 
DEBU[0009] Adding manifest "ibmcom/ashank-s390x" -> to be pushed to "ibmcom/ashank" as a manifest reference 
DEBU[0009] Manifest list push url: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test 
DEBU[0009] mediaType of manifestList: application/vnd.docker.distribution.manifest.list.v2+json 
DEBU[0010] Mount of blob sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:da86e6ba6ca197bf6bc5e9d900febd906b133eaa4750e6bed647b0fbe50ed43e" 
DEBU[0011] Mount of blob sha256:7675586df687972b960134ddaf042c570c895bf1fbdf9fc0ce0bf09c1e1c2811 succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:7675586df687972b960134ddaf042c570c895bf1fbdf9fc0ce0bf09c1e1c2811" 
DEBU[0011] Mount of blob sha256:1652adb11bd5c0bc4541f6cf812d111180a21bf88dc93ebe626ff223f103ecb8 succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:1652adb11bd5c0bc4541f6cf812d111180a21bf88dc93ebe626ff223f103ecb8" 
DEBU[0012] Mount of blob sha256:4d03cb55d8327070d0de2332365e600ad0b2afd2a30057e29cb3f9fed83e3ea0 succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:4d03cb55d8327070d0de2332365e600ad0b2afd2a30057e29cb3f9fed83e3ea0" 
DEBU[0012] Mount of blob sha256:1a792e8d3c38959ac9672a023ae365b3af7eb3a54016e2ac6a1dd0e782546cec succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:1a792e8d3c38959ac9672a023ae365b3af7eb3a54016e2ac6a1dd0e782546cec" 
DEBU[0014] Mount of blob sha256:f1f9f8769b9c22cdf3fbe980f81cc99a5ed646fb6fab74cbf5ee3f2519a98c2d succeeded, location: "https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/blobs/sha256:f1f9f8769b9c22cdf3fbe980f81cc99a5ed646fb6fab74cbf5ee3f2519a98c2d" 
DEBU[0014] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test 
DEBU[0015] referenced manifest "ibmcom/ashank-amd64" pushed; digest matches: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d 
DEBU[0015] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test 
DEBU[0016] referenced manifest "ibmcom/ashank-ppc64le" pushed; digest matches: sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0 
DEBU[0016] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test 
DEBU[0017] referenced manifest "ibmcom/ashank-s390x" pushed; digest matches: sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5 
Digest: sha256:eff4bf0e2ec1d993abc6049495795b01f0ecfbc7c6686dd148f2e1402e7ef450 1051

And the structure that I see on Artifactory is as follows. In this case, it seems like the s390x image manifest and layers are directly pushed under the test tag, along with the multi-arch manifest. The layers for the amd and ppc images are not found under this multi-arch repo at all.

ashank
  - test
    + list.manifest.json
    + manifest.json
    + sha256__1a792e8d3c38959ac9672a023ae365b3af7eb3a54016e2ac6a1dd0e782546cec
    + sha256__f1f9f8769b9c22cdf3fbe980f81cc99a5ed646fb6fab74cbf5ee3f2519a98c2d

Looking through the debug output, the main difference I see is the manifest reference push URL in the v1.0.0 manifest-tool is pointing at the test tag for all 3 arch-specific images:

DEBU[0014] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test
DEBU[0015] referenced manifest "ibmcom/ashank-amd64" pushed; digest matches: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
DEBU[0015] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test
DEBU[0016] referenced manifest "ibmcom/ashank-ppc64le" pushed; digest matches: sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0
DEBU[0016] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/test
DEBU[0017] referenced manifest "ibmcom/ashank-s390x" pushed; digest matches: sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5

Whereas the 0.7.0 tool's push url for each arch-specific image is a tag corresponding to the sha of that image:

DEBU[0020] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
DEBU[0021] referenced manifest "ibmcom/ashank-amd64" pushed; digest matches: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
DEBU[0021] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0
DEBU[0030] referenced manifest "ibmcom/ashank-ppc64le" pushed; digest matches: sha256:916a0451233bf772822b9df96e6e4c77fd2d12d76e3550b2a9fd5e5dae8f31b0
DEBU[0030] manifest reference push URL: https://hyc-cloud-private-integration-docker-local.artifactory.swg-devops.com/v2/ibmcom/ashank/manifests/sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5
DEBU[0031] referenced manifest "ibmcom/ashank-s390x" pushed; digest matches: sha256:8c8f6b6587cceaa3b3d5da351358540cf4025acad670cc7404c5b5e4fc6d45f5

So it seems like what is happening is the v1.0.0 version of the tool is uploading all the arch-specific images to the same place and they are overwriting each other (under the manifest-list tag), and then it creates a manifest list that attempts to point at all 3 shas, but only the final one pushed still exists.

[estesp]

@ashank07 you are correct--that does seem to be a bug in the migration to the newer handlers for named references between 0.7.0 and 1.0.x; interestingly the Docker v2 registry doesn't seem to care, and I know other registries seem to be handling this without an error, even though it definitely seems wrong to be pushing the sha256 references to the tag endpoint. I can inspect and even look at the on-disk content and see that somehow the Docker registry was OK with this; not because it didn't overwrite the tag, but somehow it is resolving the referenced sha's to resources which are still on disk under that image reference; which may be more of luck than design on my part, clearly.

Let me try and fix this in the 1.0.x line. I don't have access to Artifactory so may need some help testing a fix.

[estesp]

@hmtai and/or @ashank07: are you able to try out the fix in PR #100? I believe this is the problem you are hitting

[estesp]

ping @hmtai or @ashank07 : any chance you can try the above fix? Thanks!

[hmtai]

Thanks @estesp @ashank07 this fix works now.