manifest-tool seems to be missing support for private registries running on http

[estesp]

manifest-tool seems to be missing support for private registries running on http

[estesp]

Been a busy week; sorry to just now get back to you @KurtStam

Looks like the query of digest codepath was OK with HTTP, but the push/create path is not. I can look into solving this, but also note we are starting work in Docker to make manifest manipulation (including manifest list creation) part of the official docker client; so existence of this tool long-term is tenuous and dependent on my ability to maintain :)

[ekristen]

Thanks.

[aitorhh]

I have the same problem as @KurtStam with my private repository. I was trying to do the same as the option in the /etc/docker/daemon.json:

cat /etc/docker/daemon.json
{
        "insecure-registries": ["x.x.x.x:5000"]
}

I try some things like:

createml.go@func setupRepo
options.InsecureRegistries = append(options.InsecureRegistries, "x.x.x.x:5000")

or

createml.go@getHTTPClient
endpoint.TLSConfig.InsecureSkipVerify = true

But none of these worked.

@estesp is there a solution?

In the meantime I will create a self-signed certificate for my server, and try again.

Thanks!

[maitredede]

Hi,
Same problem : I have to deal with an insecure registry that works only in http (and I can't activate https), docker pull/push works fine. But neither docker manifest commands nor your tool works.

[clnperez]

there was just a fix put into the manifest command from the docker cli regarding https. you wouldn't happen to be using rhel/centos would you @maitredede ? docker/cli#1378

Edit: http/https, not just https.

[ekristen]

Bump. This is still an issue, probably just need an --insecure flag added for push.

[estesp]

Better late than never? I've opened #70 to correct this issue and will validate/merge it prior to releasing a 1.0.0-final

[ekristen]

Awesome! Definitely will help make testing mulit-arch stuff locally way easier.

[huxiaoliang]

@estesp it seems --insecure doesn't work.

1. I get the tools from wget https://github.com/estesp/manifest-tool/releases/download/v1.0.0-rc2/manifest-tool-linux-amd64

2. for the image from docker hub, it works well

[root@icp-registry1 opt]# ./manifest-tool-linux-amd64 inspect busybox
Name:   busybox (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
 * Contains 8 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4
1  Mfst Length: 527
1     Platform:
1           -      OS: linux
1           - OS Vers: 
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant: 
1           - Feature: 
1     # Layers: 1
         layer 1: digest = sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:6c84217406361b23552450c776fa929955ac4c5dbe337dd858094e7a79109003
2  Mfst Length: 527
2     Platform:
2           -      OS: linux
2           - OS Vers: 
2           - OS Feat: []
2           -    Arch: arm
2           - Variant: v5
2           - Feature: 
2     # Layers: 1
         layer 1: digest = sha256:c83038a50f6e0d7181947b4991cf3993435db7e3462c0bd13c3a4ae97d6b432c

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:ae924ca6c548e21348940acf610e8a6b61227a6f3d9ebdb1bcbc6e1ea0a51b26
3  Mfst Length: 527
3     Platform:
3           -      OS: linux
3           - OS Vers: 
3           - OS Feat: []
3           -    Arch: arm
3           - Variant: v6
3           - Feature: 
3     # Layers: 1
         layer 1: digest = sha256:ff0ca67c9bda32fa3a301324fb4c7bd54430e981a0adcf219559a2a3c73fe713

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:6b1ef683f696c503a0afb93d06684a9a70f8f793a90220eb0da569be116ce1d1
4  Mfst Length: 527
4     Platform:
4           -      OS: linux
4           - OS Vers: 
4           - OS Feat: []
4           -    Arch: arm
4           - Variant: v7
4           - Feature: 
4     # Layers: 1
         layer 1: digest = sha256:2b35d97f9c8117d50d5d2c9164acf00aa03e41d0a80f9bc2b2044e3e92fa9688

5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:93a453ec951f06e2b60269add04b7b37b03c9f5ee4362d714e5ba15d3c4c0f77
5  Mfst Length: 527
5     Platform:
5           -      OS: linux
5           - OS Vers: 
5           - OS Feat: []
5           -    Arch: arm64
5           - Variant: v8
5           - Feature: 
5     # Layers: 1
         layer 1: digest = sha256:b04ab0589b9a6d0d597a66bae318d4b08520957d4acfc7bf75496e38d3d7c8d3

6    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
6       Digest: sha256:8e2b1f48d056ac63cde2d1b7c14e35f2cc582aa7223b50ddaf24b769e6d705ec
6  Mfst Length: 527
6     Platform:
6           -      OS: linux
6           - OS Vers: 
6           - OS Feat: []
6           -    Arch: 386
6           - Variant: 
6           - Feature: 
6     # Layers: 1
         layer 1: digest = sha256:79e848d156eaf50a600bb6129f0ee47b2fa6280d25a52d99d7ee48445f186103

7    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
7       Digest: sha256:a05a8023142d1496e6c2b85101da7491b6347fd9605cdfca8b3a063dfa948748
7  Mfst Length: 528
7     Platform:
7           -      OS: linux
7           - OS Vers: 
7           - OS Feat: []
7           -    Arch: ppc64le
7           - Variant: 
7           - Feature: 
7     # Layers: 1
         layer 1: digest = sha256:628fa7149e26dcaa64b2ae1ece67309565d6f7e0a04b97f5813998b196226d80

8    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
8       Digest: sha256:8b3c63feea43a24fe7f080c3a59aa9ba036416ac62d4f86a802815b7ecd7e1bd
8  Mfst Length: 528
8     Platform:
8           -      OS: linux
8           - OS Vers: 
8           - OS Feat: []
8           -    Arch: s390x
8           - Variant: 
8           - Feature: 
8     # Layers: 1
         layer 1: digest = sha256:681c8c6f047294a46fd0f6a2da7

3. the image from my private registry without auth failed

[root@xhu-proxy1 opt]# docker pull  9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
3.1: Pulling from ibmcom-amd64/pause-amd64
Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
Status: Image is up to date for 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
[root@xhu-proxy1 opt]# /opt/manifest-tool-linux-amd64 --insecure  inspect 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
FATA[0000] Get https://9.30.160.61:5000/v2/: http: server gave HTTP response to HTTPS client 
[root@xhu-proxy1 opt]# 

4. my registry was setup by follow command:

 docker run -d  -p 5000:5000  -v /usr/local/registry:/var/lib/registry  --restart=always  --name registry  registry:2

[MrZXR]

I had the same problem with huxiaoliang

[MrZXR]

@estesp it seems --insecure doesn't work.

1. I get the tools from wget https://github.com/estesp/manifest-tool/releases/download/v1.0.0-rc2/manifest-tool-linux-amd64
2. for the image from docker hub, it works well

[root@icp-registry1 opt]# ./manifest-tool-linux-amd64 inspect busybox
Name:   busybox (Type: application/vnd.docker.distribution.manifest.list.v2+json)
Digest: sha256:954e1f01e80ce09d0887ff6ea10b13a812cb01932a0781d6b0cc23f743a874fd
 * Contains 8 manifest references:
1    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
1       Digest: sha256:f79f7a10302c402c052973e3fa42be0344ae6453245669783a9e16da3d56d5b4
1  Mfst Length: 527
1     Platform:
1           -      OS: linux
1           - OS Vers: 
1           - OS Feat: []
1           -    Arch: amd64
1           - Variant: 
1           - Feature: 
1     # Layers: 1
         layer 1: digest = sha256:fc1a6b909f82ce4b72204198d49de3aaf757b3ab2bb823cb6e47c416b97c5985

2    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
2       Digest: sha256:6c84217406361b23552450c776fa929955ac4c5dbe337dd858094e7a79109003
2  Mfst Length: 527
2     Platform:
2           -      OS: linux
2           - OS Vers: 
2           - OS Feat: []
2           -    Arch: arm
2           - Variant: v5
2           - Feature: 
2     # Layers: 1
         layer 1: digest = sha256:c83038a50f6e0d7181947b4991cf3993435db7e3462c0bd13c3a4ae97d6b432c

3    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
3       Digest: sha256:ae924ca6c548e21348940acf610e8a6b61227a6f3d9ebdb1bcbc6e1ea0a51b26
3  Mfst Length: 527
3     Platform:
3           -      OS: linux
3           - OS Vers: 
3           - OS Feat: []
3           -    Arch: arm
3           - Variant: v6
3           - Feature: 
3     # Layers: 1
         layer 1: digest = sha256:ff0ca67c9bda32fa3a301324fb4c7bd54430e981a0adcf219559a2a3c73fe713

4    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
4       Digest: sha256:6b1ef683f696c503a0afb93d06684a9a70f8f793a90220eb0da569be116ce1d1
4  Mfst Length: 527
4     Platform:
4           -      OS: linux
4           - OS Vers: 
4           - OS Feat: []
4           -    Arch: arm
4           - Variant: v7
4           - Feature: 
4     # Layers: 1
         layer 1: digest = sha256:2b35d97f9c8117d50d5d2c9164acf00aa03e41d0a80f9bc2b2044e3e92fa9688

5    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
5       Digest: sha256:93a453ec951f06e2b60269add04b7b37b03c9f5ee4362d714e5ba15d3c4c0f77
5  Mfst Length: 527
5     Platform:
5           -      OS: linux
5           - OS Vers: 
5           - OS Feat: []
5           -    Arch: arm64
5           - Variant: v8
5           - Feature: 
5     # Layers: 1
         layer 1: digest = sha256:b04ab0589b9a6d0d597a66bae318d4b08520957d4acfc7bf75496e38d3d7c8d3

6    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
6       Digest: sha256:8e2b1f48d056ac63cde2d1b7c14e35f2cc582aa7223b50ddaf24b769e6d705ec
6  Mfst Length: 527
6     Platform:
6           -      OS: linux
6           - OS Vers: 
6           - OS Feat: []
6           -    Arch: 386
6           - Variant: 
6           - Feature: 
6     # Layers: 1
         layer 1: digest = sha256:79e848d156eaf50a600bb6129f0ee47b2fa6280d25a52d99d7ee48445f186103

7    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
7       Digest: sha256:a05a8023142d1496e6c2b85101da7491b6347fd9605cdfca8b3a063dfa948748
7  Mfst Length: 528
7     Platform:
7           -      OS: linux
7           - OS Vers: 
7           - OS Feat: []
7           -    Arch: ppc64le
7           - Variant: 
7           - Feature: 
7     # Layers: 1
         layer 1: digest = sha256:628fa7149e26dcaa64b2ae1ece67309565d6f7e0a04b97f5813998b196226d80

8    Mfst Type: application/vnd.docker.distribution.manifest.v2+json
8       Digest: sha256:8b3c63feea43a24fe7f080c3a59aa9ba036416ac62d4f86a802815b7ecd7e1bd
8  Mfst Length: 528
8     Platform:
8           -      OS: linux
8           - OS Vers: 
8           - OS Feat: []
8           -    Arch: s390x
8           - Variant: 
8           - Feature: 
8     # Layers: 1
         layer 1: digest = sha256:681c8c6f047294a46fd0f6a2da7

1. the image from my private registry without auth failed

[root@xhu-proxy1 opt]# docker pull  9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
3.1: Pulling from ibmcom-amd64/pause-amd64
Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d
Status: Image is up to date for 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
[root@xhu-proxy1 opt]# /opt/manifest-tool-linux-amd64 --insecure  inspect 9.30.160.61:5000/ibmcom-amd64/pause-amd64:3.1
FATA[0000] Get https://9.30.160.61:5000/v2/: http: server gave HTTP response to HTTPS client 
[root@xhu-proxy1 opt]# 

1. my registry was setup by follow command:

 docker run -d  -p 5000:5000  -v /usr/local/registry:/var/lib/registry  --restart=always  --name registry  registry:2

I solved this problem temporarily by adding temporary code.
in docker/createml.go, line 336
endpoints, err := registryService.LookupPushEndpoints(reference.Domain(repoInfo.Name)) endpoints = endpoints[1:]

for docker/inspect.go, also modify endpoints.

this worked for me.
I'm sorry for the ugly code because I haven't used golang before T_T

[huxiaoliang]

@MrZhaoAtBJ could you help create pr to fix this issue?

[MrZXR]

@huxiaoliang sorry I cannot. I am fresh new to this field and I do not have enough ability to write golang code for production environment T_T

[estesp]

Let me take a look; I had tried several commands with a private registry on port 5000 so I thought it was working properly. The "hack" to just ignore the first entry (which is https I assume) and go to the http-only entry needs to be a more complete "fallback" mechanism that tries the endpoint entries in order. Should be fixable.

[MrZXR]

Yeah, I added some debug log, first entry is https. Maybe can skip trying https entry.

[estesp]

See #77 for a more complete fix