💸 Automatic startup and shutdown for Kafka host

[terkwood]

💸 Automatic startup and shutdown for Kafka host

[Terkwood]

Related: #95

[Terkwood]

We don't need a Kafka streams app to track Liveness of the system. Just a rust app which uses rdkafka and keeps a count in memory.

[Terkwood]

https://docs.rs/rusoto_ec2/0.40.0/rusoto_ec2/struct.StopInstancesRequest.html

[Terkwood]

https://coreos.com/os/docs/latest/customizing-docker.html

[Terkwood]

https://coreos.com/os/docs/latest/mounting-storage.html

[Terkwood]

Try a startup script

https://gist.github.com/Terkwood/7a01ad35f500934baf4da8a9dd2f8767

[Terkwood]

https://github.com/coreos/docs/blob/master/os/getting-started-with-systemd.md

[Terkwood]

confluentinc/kafka-streams-examples#180

[Terkwood]

https://docs.confluent.io/current/streams/faq.html#interactive-queries

[Terkwood]

gateway after big box restart

thread '<unnamed>' panicked at 'Nested error (!) waiting on kafka stream: KafkaError (Message consumption error: GroupLoadInProgress (Broker: Group coordinator load in progress))', src/kafka_io.rs:111:27
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
thread '<unnamed>' panicked at 'Unable to receive kafka event via router channel: RecvError', src/router.rs:275:29

[Terkwood]

orderly shutdown detected

kafkacat -b friend:9092 -t bugout-shutdown-ev -C -K:

...

1574821076618:{"secs_since_epoch":1574821076,"nanos_since_epoch":618310524}

[Terkwood]

Since we plan to make gateway stateless using Redis, it seems reasonable to use Redis for pub sub to bugle /startup service

Connection management: https://github.com/sfackler/r2d2

[Terkwood]

Hopefully we don't need this, but: rusoto doc for private DNS name of an instance

https://rusoto.github.io/rusoto/rusoto_ec2/struct.Instance.html#structfield.private_dns_name

[Terkwood]

https://stackoverflow.com/questions/35878270/docker-redis-container-getting-warning-proc-sys-net-core-somaxconn-in-a-user-d