containerized poseidon: memory corruption

[kubernetes-retired]

containerized poseidon: memory corruption

[dalanlan]

And I do apologize for not trying hard enough to bring down kube-scheduler, which could cause some chaos here. That's because it's guarded by docker daemon, and a simple docker stop won't help much.

[ms705]

@dalanlan Thanks for reporting this! The memory corruption definitely looks scary, let's try to get to the bottom of this.

A couple of questions on your setup:

1. Did you build the Poseidon image yourself, or did you download the image from DockerHub?
2. What cluster setup and workload are you running? I can see some k8s internal pods in the log, but also busybox. Are all these pods pre-existing, or did you submit new ones during this run?
3. Is the bug deterministic, i.e., does is happen the same way every time?

Moreover, there seem to be several problems here:

1. The memory corruption, obviously.
2. The fact that Poseidon fails to bind the pods because it receives a 404 error from the k8s API server when trying to do so.

The second issue could be related to kube-scheduler also running in the environment, but the memory corruption ought not be related.

The best way to debug the memory corruption is to make a debug build of Poseidon. To do this, follow these steps in a blank checkout:

$ mkdir build
$ cd build
$ cmake .. -DDEBUG=1
$ make

This will take a while, but should build Poseidon, Firmament and all of their dependencies with debug symbols and address sanitizers.

Once done, run your experiment again and see if it produces more useful output when the memory corruption happens.

If that doesn't help, you can run Poseidon in gdb:

$ docker run camsas/poseidon:dev /bin/bash
container# apt-get update
container# apt-get install build-essential gdb
container# gdb --args /usr/bin/poseidon \
    --logtostderr \
    --k8s_apiserver_host=<host> \
    --k8s_apiserver_port=<port> \
    --cs2_binary=/usr/bin/cs2.exe \
    --max_tasks_per_pu=<max pods per node>

This will run Poseidon in gdb and should give you more information at the point of crashing.

[dalanlan]

@ms705 Thank you so much for the quick response!

Actually I tried to restart the container again and it didn't crash until now. However I was thinking it could help somehow to expose more information here, which I should do long before.

docker version: 1.12.3
k8s version: 1.4.5
poseidon: camsas/poseidon:dev (built 6 months ago)

I tried to build my own image from the HEAD several days ago, but one thing that definitely make my life miserable is that I have to struggle with the poor networking in my lab all the time.

I've rather leave this thread open until I get luck to reproduce.

[ms705]

@dalanlan Ok, glad it works better now!

This reminds me, however, that we haven't pushed an updated Docker image in a while, even though things have advanced. I'll build and push one now.

[ms705]

New Docker image pushed to camsas/poseidon:dev; please give it a try.

[dalanlan]

@ms705 Thank you so much
Any luck you could take a look at #2 please?

[ms705]

Closing, as no longer relevant -- Poseidon has moved to Go, and the docker images have been updated.

Let us know if any questions remain, or if you encounter issues with the new code.