Ironic container is not very kubernetes friendly

[metal3-io]

Ironic container is not very kubernetes friendly

[dtantsur]

Hi Ian, great points!

We can't bind mount anything in, so I've got a PR to download the images at startup.

Is there any k8s-friendly way to push blobs to a container? If not, downloading may be fine.

We shouldn't really be running 4 processes in one container.

I fully agree. But it was an explicit design request. I'm totally open to going back to how TripleO/Kolla splits the containers. This will need switching to MySQL, but that's on our roadmap anyway. Also note that some containers will need a shared volume (the same way TripleO does it).

Ideally we would have 1 process per container logging to stdout so we can get logs from kubernetes.

Yeah, I think using /var/log is a side-effect from having 1 container. Also note that the current containers are more or less hacked together to Just Work.

httpd is listening on port 80?

We can change to anything.

[imain]

Hi Ian, great points!

We can't bind mount anything in, so I've got a PR to download the images at startup.

Is there any k8s-friendly way to push blobs to a container? If not, downloading may be fine.

We may be able to use a volume in the future, but I think our devel setup atm does not have any storage options to back the volumes. I think for now downloading make sense.

We shouldn't really be running 4 processes in one container.

I fully agree. But it was an explicit design request. I'm totally open to going back to how TripleO/Kolla splits the containers. This will need switching to MySQL, but that's on our roadmap anyway. Also note that some containers will need a shared volume (the same way TripleO does it).

Yeah, not a big rush on this one. Thanks! :)

Ideally we would have 1 process per container logging to stdout so we can get logs from kubernetes.

Yeah, I think using /var/log is a side-effect from having 1 container. Also note that the current containers are more or less hacked together to Just Work.

Yeah I hear ya :)

httpd is listening on port 80?

We can change to anything.

Awesome, thanks!

[hardys]

FWIW the reason for the monolithic container was to make it easy to spin up a container on the host for bootstrapping the masters - recent versions of podman do support starting a pod yaml via podman play, so we could potentially move to multi-container using that (although last time I tried it it didn't seem to work).

So we have to balance debugging convenience vs potential inconvenience of juggling multiple containers outside of the k8s environment I think. If we can make it simple to launch the pod on the host for bootstrapping then I'm fine to split the container up, but I agree it's probably not a super-high priority at this point?

@derekhiggins may have thoughts on this as well

[imain]

We could actually reuse the same container and just have multiple entry points. This would require proper healthchecks for each service though.

[imain]

At any rate, this isn't the top priority. Moving the port and getting it to work with both podman and openshift is really the first thing that needs to be solved.

[bfournie]

#16 is the first pass of splitting up containers. It became necessary to land this sooner rather than later in order to allow dnmasq to be cleanly stopped or modified on the host when the BMO is provisioning worker nodes.

I agree that we should look at podman play and podman generate to use yaml files for the podman definition and make it more compatible wit k8s. The version of podma/libpod installed with k8s doesn't have support for these commands so we'd need to upgrade.

[elfosardo]

FYI the version of podman installed in CentOS 7.6 supports "play" and "generate" subcommands

lsb_release -rd
Description: CentOS Linux release 7.6.1810 (Core) 
Release: 7.6.1810

podman --version
podman version 1.0.2-dev

sudo podman info --debug
debug:
  compiler: gc
  git commit: ""
  go version: go1.10.2
  podman version: 1.0.2-dev
host:
  BuildahVersion: 1.6-dev
  Conmon:
    package: podman-1.0.0-3.git921f98f.el7.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.14.0-dev, commit: a317801126b0e1d7d171d84dc370b98cf21fbda4-dirty'
  Distribution:
    distribution: '"centos"'
    version: "7"
  MemFree: 18512162816
  MemTotal: 33513533440
  OCIRuntime:
    package: runc-1.0.0-60.dev.git2abd837.el7.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.0'
  SwapFree: 16844320768
  SwapTotal: 16844320768
  arch: amd64
  cpus: 16
  hostname: host12.beaker.tripleo.lab.eng.rdu2.redhat.com
  kernel: 3.10.0-957.10.1.el7.x86_64
  os: linux
  rootless: false
  uptime: 32h 24m 2.34s (Approximately 1.33 days)
insecure registries:
  registries: []
registries:
  registries:
  - registry.access.redhat.com
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
  ImageStore:
    number: 12
  RunRoot: /var/run/containers/storage

sudo podman play
NAME:
   podman play - Play a pod and its containers from a structured file.

USAGE:
   podman play command [command options] [arguments...]

COMMANDS:
     kube  Play a pod based on Kubernetes YAML

OPTIONS:
   --help, -h  show help

sudo podman generate
NAME:
   podman generate - generate structured data based for a containers and pods

USAGE:
   podman generate command [command options] [arguments...]

COMMANDS:
     kube  Generate Kubernetes pod YAML for a container or pod

OPTIONS:
   --help, -h  show help

[hardys]

So we're now launching separate containers via dev-scripts, but not yet using podman play - @imain can you please help identify the existing todo items so we can work towards closing out this issue?

[elfosardo]

Just some up-to-date notes from a quick research based on version 1.2.0

Seems the generate command is quite limited at the moment, with no support for dependencies (volumes, other containers), so automated generation of yaml files is no go.

The play command works ok, although some limitations there as well, for example:

• only one type of volume supported
  https://github.com/containers/libpod/blob/master/cmd/podman/play_kube.go#L157
• security policies can be defined only on container level and not on pod level
• full privileged containers are still granted limited Linux capabilities

I'm not sure we can use the play command for what we need in dev-scripts at the moment, considering we allow very open security policy.

rpittau@host12:~ $ lsb_release -rd
 Description: CentOS Linux release 7.6.1810 (Core) 
 Release: 7.6.1810

rpittau@host12:~ $ podman --version
 podman version 1.2.0

rpittau@host12:~ $ sudo podman info --debug
 debug:
 compiler: gc
 git commit: ""
 go version: go1.10.2
 podman version: 1.2.0
 host:
 BuildahVersion: 1.7.2
 Conmon:
 package: podman-1.2-2.git3bd528e.el7.x86_64
 path: /usr/libexec/podman/conmon
 version: 'conmon version 1.14.0-dev, commit: 345710c5d359e8d5b126906e24615d6a3e28c131-dirty'
 Distribution:
 distribution: '"centos"'
 version: "7"
 MemFree: 25076391936
 MemTotal: 33513533440
 OCIRuntime:
 package: runc-1.0.0-60.dev.git2abd837.el7.x86_64
 path: /usr/bin/runc
 version: 'runc version spec: 1.0.0'
 SwapFree: 16838029312
 SwapTotal: 16844320768
 arch: amd64
 cpus: 16
 hostname: host12.beaker.tripleo.lab.eng.rdu2.redhat.com
 kernel: 3.10.0-957.10.1.el7.x86_64
 os: linux
 rootless: false
 uptime: 508h 43m 23.67s (Approximately 21.17 days)
 insecure registries:
 registries: []
 registries:
 registries:
 - registry.access.redhat.com
 - docker.io
 - registry.fedoraproject.org
 - quay.io
 - registry.centos.org
 store:
 ConfigFile: /etc/containers/storage.conf
 ContainerStore:
 number: 0
 GraphDriverName: overlay
 GraphOptions: null
 GraphRoot: /var/lib/containers/storage
 GraphStatus:
 Backing Filesystem: xfs
 Native Overlay Diff: "true"
 Supports d_type: "true"
 Using metacopy: "false"
 ImageStore:
 number: 30
 RunRoot: /var/run/containers/storage
 VolumePath: /var/lib/containers/storage/volumes

[imain]

It sounds to me like it wouldn't be worth moving to play. I don't see a problem with the current set up in dev-scripts and the kubernetes pod I'm working on. They are different beasts and I don't think any code would likely to be shared there.

You guys did a great job on making the containers configurable via the entrypoint/env variables! Thank you!

Really the only thing left is to implement the health checks properly, which would require a lot more podman wrangling. I'll open a separate issue for this though.

Nice work! Thanks guys!