financial-times/content-k8s-provisioner

content-k8s-provisioner's Issues

Restore the k8s config from backup in provisioner

Provisioning a new cluster should restore the config from the S3 bucket automatically.

https://trello.com/c/sTUINNZz/48-restore-the-k8s-config-from-backup-in-provisioner

Generate proper TLS keys for our k8s clusters

From the kube-aws docs:

PRODUCTION NOTE: the TLS keys and certificates generated by kube-aws should not be used to deploy a production Kubernetes cluster. Each component certificate is only valid for 90 days, while the CA is valid for 365 days. If deploying a production Kubernetes cluster, consider establishing PKI independently of this tool first. Read more below.

https://kubernetes-incubator.github.io/kube-aws/getting-started/step-2-render.html#render-cluster-assets

---

https://trello.com/c/qq0XofH8/16-generate-proper-tls-keys-for-our-k8s-clusters

Put the needed etcd keys for Splunk

https://trello.com/c/TaYRs7CC/107-provisioner-put-the-needed-etcd-keys-for-splunk

Pass Konsturctor key as env variable

The only key that is stored in the ansible vault is the konstructor dns key.
When a key needs rotation, it is changed in the ansible vault file and a PR is submitted.

To avoid the hassle of submitting PR's, lets pass the key as an env variable and the only place the key has to be update will be the provisioner last pass note.

Making this change in the provisioner would also involve making changes to the jenkins pipeline:

• Remove reference to vault pass and set the konsturctor dns key https://github.com/Financial-Times/k8s-pipeline-library/blob/master/src/com/ft/jenkins/provision/ProvisionerUtil.groovy
• Create credentials for konsturctor in jenkins

Run alertlogic as a systemd service on the etcd cluster

https://trello.com/c/OWvIclDD/30-run-alertlogic-as-a-systemd-service-on-the-etcd-cluster

Update SQS access to content-container-apps user based on environment

We have content-container-apps user for Prod and Staging

https://github.com/Financial-Times/upp-provisioners/blob/master/upp-concept-publishing-provisioner/cloudformation/sns-sqs-cf-multi-region.yml#L57

The permission on the SQS policy needs to be set based on environment

Prod: content-container-apps
Staging: content-container-apps-staging

https://trello.com/c/nf3IIPuU/122-provisioner-update-sqs-access-to-content-container-apps-user-based-on-environment

Generate cluster IDs

Generate cluster IDs for each region of each environments, used by notifications-rw.

Has to be
- unique per region
- in CoCo comes from the first few digits of the etcd discovery URL

3590c90

Or find out what notifications-rw needs it for and possibly change it.

https://trello.com/c/aKSShASV/83-generate-cluster-ids

Auto-scaling for workers (not essential)

• in order to cope better with high loads, it would be useful to scale up the worker pool under higher loads
• we could live without this, but we would need bigger machines, which could handle spikes in load versus smaller machines with load-based auto-scaling

https://trello.com/c/n9CUx3U7/92-auto-scaling-for-workers-not-essential

Keep journald logs locally on the instance for more time (2d)

Right now we're running with the default journald settings that rotates the logs based on size.
We should keep them at least 2d.

https://trello.com/c/8YZSzco3/156-keep-journald-logs-locally-on-the-instance-for-more-time-2d

Discuss if we need in provisioner a "reprovision only k8s" task

https://trello.com/c/ox8PeUk7/109-provisioner-discuss-if-we-need-in-provisioner-a-reprovision-only-k8s-task

UPP Elastic Search

Now that we have different content-container-apps for prod and staging
content-container-apps for Prod
content-container-apps-staging for Staging

the access policy for the ES cluster should be automatically set based on the environment

https://github.com/Financial-Times/upp-provisioners/blob/master/upp-elasticsearch-provisioner/cloudformation/upp-concepts.yml#L49

https://trello.com/c/Sc68VBMu/121-provisioner-upp-elastic-search

Githubify PAC's global config

Move PAC global config to GitHub from the cluster itself.

https://trello.com/c/MtqIVtfG/29-githubify-pacs-global-config

Provision the S3 for publish-carousel

We recently find out that the publish-carousel needs its own env specific S3 bucket

https://trello.com/c/Smaow3HC/97-provisioner-provision-the-s3-for-publish-carousel

On decom delete also the ELB used as entry point by the cluster (by the varnish services)

https://trello.com/c/DnvUUqbd/120-provisioner-on-decom-delete-also-the-elb-used-as-entry-point-by-the-cluster-by-the-varnish-services