panic when running docker-compose up

[concourse]

panic when running docker-compose up

[dzirg44]

I rewrited keygen script because in modern systems ssh-keygen generated keys didn't work property with golang.org/x/crypto/ssh package. I will try make patch for this on this weekend if i will have time.

#!/usr/bin/env bash

set -e -u -x

mkdir -p keys/web keys/worker

openssl genpkey -algorithm RSA -out ./keys/web/tsa_host_key -pkeyopt rsa_keygen_bits:4096
ssh-keygen -y -f  ./keys/web/tsa_host_key > ./keys/web/tsa_host_key.pub

openssl genpkey -algorithm RSA -out ./keys/web/session_signing_key -pkeyopt rsa_keygen_bits:4096
ssh-keygen -y -f  ./keys/web/session_signing_key > ./keys/web/session_signing_key.pub


openssl genpkey -algorithm RSA -out ./keys/worker/worker_key -pkeyopt rsa_keygen_bits:4096
ssh-keygen -y -f  ./keys/worker/worker_key > ./keys/worker/worker_key.pub

cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
cp ./keys/web/tsa_host_key.pub ./keys/worker

It is a temporary solution.

[danielrs]

I created a pull request to fix this issue (#30). OpenSSH has been generating keys in a different format since 7.8. Check here for more info.

[whi-tw]

Also experiencing this with MacOS - seems to be some kind of weirdness with ssh-keygen.

Resolved with docker run --rm -v$(pwd):$(pwd) -w $(pwd) ubuntu:latest bash -c 'apt-get update && apt-get install -y openssh-client && ./generate-keys.sh'

[cirocosta]

Hey @sizgiyaev,

It might be that you missed the step regarding key generation.

From the readme:

There are two Docker Compose .yml files in this repo. The first one, docker-compose.yml, runs a more traditional multi-container cluster. You'll need to run ./generate-keys.sh before booting up so that the containers know how to authorize each other.

So, that'd mean that ./generate-keys.sh would be required to be run before docker-compose up.

Please let me know if that helps!

Thx

[ahaczewski]

@cirocosta That is not it, I've run ./generate-keys.sh as well as sudo ./generate-keys.sh (so the user does not matter) and both times it ends with that panic, just after ATC rotates expired keys.

Here's full stacktrace with 2 important messages from ATC:

{"timestamp":"1537295512.434964895","source":"atc","message":"atc.dex.event","log_level":1,"data":{"fields":{},"message":"keys expired, rotating","session":"5"}}
{"timestamp":"1537295512.620816469","source":"atc","message":"atc.dex.event","log_level":1,"data":{"fields":{},"message":"keys rotated, next rotation: 2018-09-19 00:31:52.618429464 +0000 UTC m=+21641.400663850","session":"5"}}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe3485]

goroutine 1 [running]:
crypto/rsa.(*PrivateKey).Public(0x0, 0x0, 0x0)
	/usr/local/go/src/crypto/rsa/rsa.go:100 +0x5
golang.org/x/crypto/ssh.NewSignerFromSigner(0x7eff3843c308, 0xc00068e760, 0xc00068e760, 0x7eff3843c308, 0xc00068e760, 0xe8af01)
	/tmp/build/9674af12/concourse/src/golang.org/x/crypto/ssh/keys.go:681 +0x35
golang.org/x/crypto/ssh.NewSignerFromKey(0x2a2eb40, 0xc00068e760, 0xe8afc8, 0x30, 0x2b17180, 0xc000a01408)
	/tmp/build/9674af12/concourse/src/golang.org/x/crypto/ssh/keys.go:656 +0x16a
github.com/concourse/tsa/tsacmd.(*TSACommand).configureSSHServer(0xc0001d2e70, 0xc000767220, 0xc0002f6750, 0x1, 0x1, 0x0, 0x0, 0x0, 0xc000198380, 0x7, ...)
	/tmp/build/9674af12/concourse/src/github.com/concourse/tsa/tsacmd/command.go:197 +0x167
github.com/concourse/tsa/tsacmd.(*TSACommand).Runner(0xc0001d2e70, 0xc0002f6420, 0x0, 0x1, 0x17dbe900, 0xc0001f3980, 0x0, 0x0)
	/tmp/build/9674af12/concourse/src/github.com/concourse/tsa/tsacmd/command.go:95 +0x249
main.(*WebCommand).Runner(0xc000805808, 0xc0002f6420, 0x0, 0x1, 0x2, 0x5, 0xc0002f62b0, 0xc00067dcc0)
	/tmp/build/9674af12/concourse/src/github.com/concourse/bin/cmd/concourse/web.go:67 +0x132
main.(*WebCommand).Execute(0xc000805808, 0xc0002f6420, 0x0, 0x1, 0x28fde00, 0x2b848a0)
	/tmp/build/9674af12/concourse/src/github.com/concourse/bin/cmd/concourse/web.go:42 +0x5c
github.com/vito/twentythousandtonnesofcrudeoil.installEnv.func2(0x7eff2c02bae8, 0xc000805808, 0xc0002f6420, 0x0, 0x1, 0x1, 0xc000ccb7e8)
	/tmp/build/9674af12/concourse/src/github.com/vito/twentythousandtonnesofcrudeoil/environment.go:40 +0x8a
github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc00005a1e0, 0xc00000c070, 0x1, 0x1, 0xc000ccb880, 0x20fe6bf, 0xc00005a1e0, 0x2bc8013, 0xa)
	/tmp/build/9674af12/concourse/src/github.com/jessevdk/go-flags/parser.go:314 +0x7bc
github.com/jessevdk/go-flags.(*Parser).Parse(0xc00005a1e0, 0x2bc8013, 0xa, 0xc00066b580, 0x0, 0x0)
	/tmp/build/9674af12/concourse/src/github.com/jessevdk/go-flags/parser.go:186 +0x71
main.main()
	/tmp/build/9674af12/concourse/src/github.com/concourse/bin/cmd/concourse/main.go:33 +0x19e

[sizgiyaev]

it helped me
Thanks a lot

[ahaczewski]

Great, thanks for the workaround.

[loghen41]

I have also seen this issue on fedora, @tnwhitwell solution was simple and lightweight to get those keys generated easliy.

[kwladyka]

I rewrited keygen script because in modern systems ssh-keygen generated keys didn't work property with golang.org/x/crypto/ssh package

Why no update concourse to work with it?

[larssb]

@danielrs merge request has been merged. @sizgiyaev it seems to be working for you. So could you come back and close this issue if you are happy go lucky 👍

Cool and have a great one.

[johnewhitley]

I don't think that fixed the key issue.

concourse-web_1     | {"timestamp":"1547062866.916333675","source":"atc","message":"atc.dex.event","log_level":1,"data":{"fields":{},"message":"keys expired, rotating","session":"5"}}
concourse-web_1     | {"timestamp":"1547062867.185469866","source":"atc","message":"atc.dex.event","log_level":1,"data":{"fields":{},"message":"keys rotated, next rotation: 2019-01-10 01:41:07.182983567 +0000 UTC m=+21602.501571442","session":"5"}}
concourse-web_1     | panic: runtime error: invalid memory address or nil pointer dereference
concourse-web_1     | [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xfe3485]
concourse-web_1     |
concourse-web_1     | goroutine 1 [running]:
concourse-web_1     | crypto/rsa.(*PrivateKey).Public(0x0, 0x0, 0x0)
concourse-web_1     | 	/usr/local/go/src/crypto/rsa/rsa.go:100 +0x5
concourse-web_1     | golang.org/x/crypto/ssh.NewSignerFromSigner(0x7fa59e1a9500, 0xc00017c668, 0xc00017c668, 0x7fa59e1a9500, 0xc00017c668, 0xe8af01)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/golang.org/x/crypto/ssh/keys.go:681 +0x35
concourse-web_1     | golang.org/x/crypto/ssh.NewSignerFromKey(0x2a2eb40, 0xc00017c668, 0xe8afc8, 0x30, 0x2b17180, 0xc00085d408)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/golang.org/x/crypto/ssh/keys.go:656 +0x16a
concourse-web_1     | github.com/concourse/tsa/tsacmd.(*TSACommand).configureSSHServer(0xc000e5b340, 0xc000475e70, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc0001b2380, 0x7, ...)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/concourse/tsa/tsacmd/command.go:197 +0x167
concourse-web_1     | github.com/concourse/tsa/tsacmd.(*TSACommand).Runner(0xc000e5b340, 0xc0001677c0, 0x0, 0x1, 0x17dbe900, 0xc000685d60, 0x0, 0x0)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/concourse/tsa/tsacmd/command.go:95 +0x249
concourse-web_1     | main.(*WebCommand).Runner(0xc0001a8308, 0xc0001677c0, 0x0, 0x1, 0x2, 0x5, 0xc000167650, 0xc000d27540)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/concourse/bin/cmd/concourse/web.go:67 +0x132
concourse-web_1     | main.(*WebCommand).Execute(0xc0001a8308, 0xc0001677c0, 0x0, 0x1, 0x28fde00, 0x2b848a0)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/concourse/bin/cmd/concourse/web.go:42 +0x5c
concourse-web_1     | github.com/vito/twentythousandtonnesofcrudeoil.installEnv.func2(0x7fa59e2482e0, 0xc0001a8308, 0xc0001677c0, 0x0, 0x1, 0x1, 0xc000f137e8)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/vito/twentythousandtonnesofcrudeoil/environment.go:40 +0x8a
concourse-web_1     | github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc000168300, 0xc0000d0010, 0x1, 0x1, 0xc000f13880, 0x20fe6bf, 0xc000168300, 0x2bc8013, 0xa)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/jessevdk/go-flags/parser.go:314 +0x7bc
concourse-web_1     | github.com/jessevdk/go-flags.(*Parser).Parse(0xc000168300, 0x2bc8013, 0xa, 0xc000700d00, 0x0, 0x0)
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/jessevdk/go-flags/parser.go:186 +0x71
concourse-web_1     | main.main()
concourse-web_1     | 	/tmp/build/9674af12/concourse/src/github.com/concourse/bin/cmd/concourse/main.go:33 +0x19e
concourse-db_1      | 2019-01-09 19:41:07.244 UTC [35] LOG:  could not send data to client: Broken pipe
concourse-db_1      | 2019-01-09 19:41:07.244 UTC [35] FATAL:  connection to client lost
concourse-docker_concourse-web_1 exited with code 2

[avoidik]

@dzirg44 suggested had helped me to solve the issue, don't forget to re-create docker-machine if you're mapping generated keys as volumes