How to deploy a docker container to otc?

[opentelekomcloud]

How to deploy a docker container to otc?

[excurs0r]

Got mail from Telekom. It's not possible. There is a "Cloud Container Engine" with documentation explaining how to install certificates, tag docker images, login with docker and push them to the registry. But at the moment this part is broken.

[tischrei]

I believe, there is a misunderstanding, because I am pretty sure that I am the one who wrote you the mail, so I can clear this up.
The initial question was: "How to deploay docker containers with terraform to Open Telekom Cloud?" In the mail was the first question from my side: Which environment is used to deploy docker containers? Terraform is only a way to create infrastructure ressources where a docker host or cluster can resides on. There is no way to create docker containers "directly" with Terraform.
The mentioned Cloud Container Engine (CCE) is one way how we provide a Platform as a Service (PaaS) for our customers to easily create a Kubernetes based docker cluster and run containers on. But there is no Terraform-interface to create such a CCE-cluster.

[excurs0r]

Nice to hear from you :) I checked it and you're right, it was your mail.
I'd really like to get things started, but I really don't know what
I do wrong.

To get things clear for everyone I'll try my best to describe,
what I've done and where I failed.

First of all, I'm working on Debian/Buster.
I've installed lots of stuff. Amongst other things
docker and terraform.

So this is how I understood the deployment process of the infrastructure:

1. Configure local machine to be able to push docker containers to the Cloud Container Engine (CCE)
2. Tag a docker image like: "127.0.0.1:9999/something/otctest"
3. Use docker with auth-token to login
4. Push docker container to registry (which is part of CCE)
5. Use terraform to setup loadbalacer etc (reference previously pushed docker image)

So what did I exactly do:

I started by logging in at https://auth.otc.t-systems.com/authui/login.action#/login.
Then I took a look at the Container Registry. There is a button that says "Upload Container Image".
I clicked on that and there were three links to documentations. I followed those. At first I created
an auth-token, configured docker with "insecure-registries" in (/etc/docker/daemon.json) and tried
to login with docker. The command I used for that was:

docker login -u _auth_token -p {generated-token} -e aa {fancy-ip}:{some-port}

This worked sometimes and sometimes not. I was unable to find out why.
Restarted docker several times (service and socket). It was like 1 out
of 15. There were no typos, because I always reused the old command via
bck-i-search (Ctrl+R).

Then I took the next steps.
I added the certificates to have trusted content. I put them in
/usr/share/ca-certificates

Then I updated certificates: sudo update-ca-certificates
Output says they were added.

I also set the environment variables:

DOCKER_CONTENT_TRUST=1
DOCKER_CONTENT_TRUST_SERVER={cool-ip}:{fancy-port}

The next step is to tag a docker image and then push it to container registry.
So I tagged an image. With "docker images" command I checked it again:
{ip}:{port}/something/someimage latest

And now comes the part, where I failed.
docker push {ip}:{port}/something/someimage

And this is the output:

The push refers to a repository [{ip}:{port}/something/someimage]
068f00f601b4: Preparing
ffb1344a0bfb: Preparing
unauthorized: authentication required

The login didn't expire. And the docker error message is not very helpful,
because unauthorized means "I know you, but you are not allowed to do that"
and authentication required means "I don't know you, please login".

So I started debugging that. First I took a look at /var/log/daemon.log.
There it says:

Feb 21 13:59:18 machine dockerd[19533]: time="2018-02-21T13:59:18.079352446+01:00" level=error msg="Upload failed: unauthorized: authentication required"                                                            │
Feb 21 13:59:18 machine dockerd[19533]: time="2018-02-21T13:59:18.079424439+01:00" level=error msg="Attempting next endpoint for push after error: unauthorized: authentication required"

I'm not 100% sure about it, but I think you use docker registry v1 and not v2. Is that right?

My next attempt was to checkout sudo journalctl -fu docker.service.
The journal tells me:

Feb 21 14:04:01 machine dockerd[19533]: time="2018-02-21T14:04:01.118711156+01:00" level=warning msg="failed to retrieve docker-runc version: unknown output format: runc version spec: 1.0.0-rc2-dev
Feb 21 14:04:01 machine dockerd[19533]: "
Feb 21 14:04:01 machine dockerd[19533]: time="2018-02-21T14:04:01.118771754+01:00" level=warning msg="failed to retrieve docker-init version"
Feb 21 14:04:05 machine dockerd[19533]: time="2018-02-21T14:04:05.433738752+01:00" level=error msg="Upload failed: unauthorized: authentication required"
Feb 21 14:04:05 machine dockerd[19533]: time="2018-02-21T14:04:05.433843550+01:00" level=error msg="Attempting next endpoint for push after error: unauthorized: authentication required"

The logs are from right now. I check everything, while writing.

There were some colleagues helping me out with stuff, but we were unable to solve it.
We even installed our own registry and there was no problem in pushing and pulling.
The "docker-init version" and "docker-runc" messages also appeared. Even though I looked
around in the internet to find a solution, I could not get rid of them. And the internet
says that they are not relevant. Our own registry confirms that.

We also tried to do this stuff with packer, which has a docker-tag and docker-push hook.
Btw packer is also unable to get a stable login. Also takes some tries.
Anyway, the output is:

packer build otctest.json

docker output will be in this color.

==> docker: Creating a temporary directory for sharing data...
==> docker: Pulling Docker image: debian:stable-slim
    docker: stable-slim: Pulling from library/debian
    docker: Digest: sha256:79fa38298b2201284248246f51327403c8075d8c0704cf0e2694668d213d5b0f
    docker: Status: Image is up to date for debian:stable-slim
==> docker: Starting docker container...
    docker: Run command: docker run -v /home/me/.packer.d/tmp/packer-docker932334302:/packer-files -d -i -t debian:stable-slim /bin/bash
    docker: Container ID: 37ec4b3083640b4d529e40b9dd9bb7136a20be0832fb0a5963c0b18d729f136c
==> docker: Committing the container
    docker: Image ID: sha256:9bd3160e75df1441d12e5a0ccfb5bb4ab6013d868a59066523a0f9462d154a38
==> docker: Killing the container: 37ec4b3083640b4d529e40b9dd9bb7136a20be0832fb0a5963c0b18d729f136c
==> docker: Running post-processor: docker-tag
    docker (docker-tag): Tagging image: sha256:9bd3160e75df1441d12e5a0ccfb5bb4ab6013d868a59066523a0f9462d154a38
    docker (docker-tag): Repository: {ip}:{port}/something/otctest:latest
==> docker: Running post-processor: docker-push
    docker (docker-push): Logging in...
    docker (docker-push): Login Succeeded
    docker (docker-push): Pushing: {ip}:{port}/something/otctest:latest
    docker (docker-push): The push refers to a repository [{ip}:{port}/something/otctest]
    docker (docker-push): e8f7eb100cd2: Preparing
    docker (docker-push): ffb1014a0bfb: Preparing
    docker (docker-push): unauthorized: authentication required
    docker (docker-push): Logging out...
    docker (docker-push): Removing login credentials for {ip}:{port}
Build 'docker' errored: 1 error(s) occurred:

* Post-processor failed: Bad exit status: 1

==> Some builds didn't complete successfully and had errors:
--> docker: 1 error(s) occurred:

* Post-processor failed: Bad exit status: 1

==> Builds finished but no artifacts were created.

Same thing here. The only thing I can think of, that is causing the failure, is
that the time on your server and the time on my machine are not identical, so the
authentication expires or somewhat. That was a problem that some people, which got
the same error messages, in the internet had with their registries. But I'm not able
to find out your server configuration. In the web interface we choosed eu-de as our
location.

I really have no clue what to do now. Hopefully those information help you or someone
to fix things. It would be nice if you could make things clear for us. Did we miss
something or made something wrong?

Thank you in advance

[28formica77]

Hi excurs0r,

I can see a couple of differences to my configuration. I have my daemon running with --insecure-registry

root 1376 1 0 2017 ? 01:09:47 /usr/bin/dockerd --log-level=info --insecure-registry 80.158.0.168:443 --containerd /run/containerd/containerd.sock

and I have a config.json wth the certificate info in it for authentication with the CCE registry.

cat ~/.docker/config.json
{"auths":{"-SNIP-:443":{"auth":"XXXXXY-SNIP-XXXXXX==","email":""}}}

Have you downloaded the certificate from Registry/+Upload Container Image?

Regards

Anthony

[tischrei]

Dear escurs0r,

at first the --insecure-registry must be configured properly in the docker host, which pulls or pushes images to the private (CCE) registry. I will try to explain it with an Ubuntu 16.04 host.

1. Add registry address

root@ecs-tino-ubuntu:~# cat /etc/docker/daemon.json
{
"insecure-registries": ["160.44.200.121:443"]
}

2. restart daemon

systemctl daemon-reload
service docker restart

3. download cert-file from CCE-Registry GUI and..
   3a)... copy the file to

root@ecs-tino-ubuntu:~# cat .docker/config.json
{
        "auths": {
                "160.44.200.121:443": {
                        "auth": "XYZ"
                }
        }
}

3b) ...or use the following command to get the _auth_token for the login process and login (safer way to check a correct registry login)

echo -n {auth} | base64 -d
docker login -u _auth_token -p {generated-token} 160.44.200.121:443

NOTE: The command -e aa {fancy-ip}:{some-port} is wrongly described and will be fixed in the next documentation review.

4. Pull image, tag the image and upload it to the registry

root@ecs-tino-ubuntu:~# docker pull httpd
Using default tag: latest
latest: Pulling from library/httpd
4176fe04cefe: Pull complete
d6c01cf91b98: Pull complete
b7066921647a: Pull complete
643378aaba88: Pull complete
3c51f6dc6a3b: Pull complete
4f25e420c4cc: Pull complete
ccdbe37da15c: Pull complete
Digest: sha256:6e61d60e4142ea44e8e69b22f1e739d89e1dc8a2764182d7eecc83a5bb31181e
Status: Downloaded newer image for httpd:latest
root@ecs-tino-ubuntu:~# docker tag httpd 160.44.200.121:443/otc00000000001000000201/httpd:latest
root@ecs-tino-ubuntu:~# docker push 160.44.200.121:443/otc00000000001000000201/httpd:latest
The push refers to a repository [160.44.200.121:443/otc00000000001000000201/httpd]
11d3a23fba24: Pushed
6702ee5815dd: Pushed
ca92f217a68e: Pushed
400eca481024: Pushed
53cbc0080070: Pushed
a9681abc377f: Pushed
ffc4c11463ee: Pushed
latest: digest: sha256:6a457fe47eaa405ea173ca61d29c4367a593e8b092ed2e6c0fda0c77d801c485 size: 1780

Please let me know if it is working for you.

Kind regards,
Tino

[excurs0r]

cat /etc/docker/daemon.json

{
"insecure-registries": ["160.44.200.121:443"]
}

cat .docker/config.json

{"auths":{"160.44.200.121:443":{"auth":"xxx","email":""}}}

sudo systemctl daemon-reload
sudo systemctl restart docker

docker login -u _auth_token -p xxx 160.44.200.121:443

Error response from daemon: Login: 404 page not found
 (Code: 404; Headers: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff] Date:[Tue, 27 Feb 2018 11:21:07 GMT] Content-Length:[19]])

[excurs0r]

Login worked with old command:

docker login -u _auth_token -p xxx  -e aa 160.44.200.121:443

Login Succeeded

[[ EDIT:
Retried docker login -u _auth_token -p xxx 160.44.200.121:443
And it worked. But still unable to get things running.
Compared login commands → No difference
]]

docker images | grep otc

160.44.200.121:443/otc00000000000000012345/otctest   latest              9bfa9917d866        12 days ago         55.3MB

docker push 160.44.200.121:443/otc00000000000000012345/otctest

The push refers to a repository [160.44.200.121:443/otc00000000000000012345/otctest]
ffb1014a0bfb: Preparing 
unauthorized: authentication require

[tischrei]

The correct "username" for registry authentication is necessary. You need to use the correct tenant ID to pull images.

e.g. my Account name (tenant ID) is: OTC00000000001000000201
The correct docker image tag / pull needs to be

docker tag {imagename:tag} 160.44.200.121:443/otc00000000001000000201/{imagename:tag}

So: docker push 160.44.200.121:443/otc00000000000000012345/otctest will not work!

Please also check ~/.docker/config.json for your user, if the login information is placed properly after the login was succesfully done.

root@ecs-tino-ubuntu:~# cat .docker/config.json
{
        "auths": {
                "160.44.200.121:443": {
                        "auth": "X2F_shortened"
                }
        }
}