kinvolk/lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk

⚠️ End of project ⚠️

This project has ended, and further Lokomotive development has halted.

As it is free software, people are free and welcome to fork and develop the codebase on their own. However, to avoid any confusion, the original repository is archived and we recommend any further fork/development to proceed with an explicit rename and rebranding first.

We encourage all interested parties to mirror any relevant bits as we can't actively guarantee their existence in the future.

---

Lokomotive

Lokomotive is an open source Kubernetes distribution that ships pure upstream Kubernetes. It focuses on being minimal, easy to use, and secure by default.

Lokomotive is fully self-hosted and includes Lokomotive Components adding essential functionality for production not shipped with upstream Kubernetes.

Features

• Kubernetes 1.19 (upstream, via kubernetes-incubator/bootkube)
• Fully self-hosted, including the kubelet
• Single or multi-master
• Calico networking
• On-cluster etcd with TLS, RBAC-enabled, network policies

Installation

Lokomotive provides the lokoctl CLI tool to manage clusters. Check the installation guide to install it.

Getting started

Follow one of the quickstart guides for the supported platforms:

• Equinix Metal quickstart
• AWS quickstart
• Bare metal quickstart

Documentation

Reference guides

• Platform configuration references
• Component configuration references
• CLI reference

How to guides

• Backup Rook Ceph volume on S3 using Velero
• Kubernetes storage with Rook Ceph on Equinix Metal
• Set up monitoring on Lokomotive
• Setting up an HTTP ingress controller on Lokomotive with MetalLB and Contour on Equinix Metal
• Setting up cluster authentication on Lokomotive with GitHub, Dex and Gangway
• Setting up third party OAuth for Grafana
• Upgrading bootstrap kubelet
• Upgrading etcd
• Using Cloudflare as a DNS provider for Lokomotive

Issues

Please file issues on this repository.

Before filing an issue, please ensure you have searched through / reviewed existing issues.

If an issue or PR you’d like to contribute to is already assigned to someone, please reach out to them to coordinate your work.

If you would like to start contributing to an issue or PR, please request to have it assigned to yourself.

Contributing

Check out our contributing guidelines.

License

Unless otherwise noted, all code in the Lokomotive repository is licensed under the Apache 2.0 license. Some portions of the codebase are derived from other projects under different licenses; the appropriate information can be found in the header of those source files, as applicable.

lokomotive's Issues

Consider using an auto scaling group for controller nodes on AWS

Using a fixed-size auto scaling group for controller nodes has the following benefits:

• Self-healing control plane. If a controller node dies or is unhealthy, it is automatically replaced by the auto scaling service. This is the main motivation for using auto scaling groups for controller nodes.

• More reliable cluster bootstrap. Auto scaling groups require us to use userdata for provisioning controllers. Creating and provisioning nodes in a single, atomic API call is inherently more stable than relying on interactive SSH sessions for machine provisioning.

Decide on enabling persistent storage by default for Grafana

While we should promote immutable approach, with fully automated dashboards (via ConfigMaps etc.), it might be annoying for the users, which create their dashboards by hand to lose them, when Grafana Pod is being removed.

We should consider enabling persistence for Grafana by default and document our recommendation:

• that we don't enable it, so users must be careful. Perhaps some reference to the guide of how to create automated dashboards will be then useful
• that we enable it for convenience, but we highly encourage users to disable it. Again we should point to the guide of how to automate them

MetalLB fails silently when the address pool is invalid

If the Address pool param for MetalLB is a map (as we expect) but, for example, the value for some address pool is not a valid CIDR (just a string, we the component expects), the MetalLB component will be installed and no error will be reported.

The same if the address pool is not a valid name (if such a restriction exists) or, maybe even if some fields are empty it will be filled with "null", etc. I mean, I think we can easily generate configmaps that are invalid for MetalLB but no error is shown on component installation.

All the other params this component has (like toleration, etc.) are validated by the kubernetes apiserver, so we don't need to validate them: we just try to install them and it will fail. But, the addess pool is different in this regard, as it is used to generate a configmap. From the kubeapiserver POV, it can be a valid configmap, but from the MetalLB POV, it can fail (MetalLB will be in running state, just log that config is invalid).

I don't think we need to duplicate the parsing logic MetalLB has, but It will be nice to not fail silently to the user.

One option, is maybe to check if MetalLB is working fine upon component installation and show the pod logs to not fail silently? It seems tricky, though. Maybe MetalLB exposes some metric in the endpoint, that we can query or something like that. We will need some investigation on what to do exactly.

Another option (way more complicated but more interesting) is to explore using CRDs for MetalLB. Upstream seems interested in that (as it is more cloud native) and in that case, the CRD should have a special format and only accept valid addressPools (don't know if the specification is reach enough to allow only valid addressPools to pass validation)

There might be other options as well :)

Set custom labels and taints for AWS platform

Description

In Lokomotive for AWS add support for custom labels and taints for worker nodes. Just like packet has.

• Custom Labels: https://github.com/kinvolk/lokomotive-kubernetes/blob/0da7e1426119a6b1539ff45a2f0adbcc7198842b/packet/flatcar-linux/kubernetes/workers/variables.tf#L37-L43
• Custom Taints: https://github.com/kinvolk/lokomotive-kubernetes/blob/0da7e1426119a6b1539ff45a2f0adbcc7198842b/packet/flatcar-linux/kubernetes/workers/variables.tf#L45-L49

Initial task breakdown

• Add the variables like it is done in Packet.
• Consume them at the Kubelet level flags.
• Make sure to port those above changes to the Lokoctl.
• Add test to lokoctl which verify that such labels/taints have been added.
• Add docs that show how to use this feature.

Acceptance Criteria

• User can set labels and taints from lokoctl and create cluster and verify that those have been applied succefully.

Follow up work

• Create similar issues/work for other supported providers (Azure, Baremetal, Google Cloud, KVM Libvirt)

Ensure all controlplane component has either liveness or readiness probes defined

Helm relies on the probes when waiting for the release to be upgraded. If probes are not defined for the workload, helm assumes that upgrades went fine and moves on. This may give false idea of successful upgrade of the cluster, which is definitely not desired, when running self-hosted controlplane.

Current matrix:

Component	Liveness/Readiness probe	Comment
Calico Node	Liveness And Readiness
Flannel	None	Do we still support it?
kube-apiserver	None
kube-router	None	Do we still support it?
kubelet	None	Does it make sense to have one with self-hosted kubelet?
CoreDNS	Liveness And Readiness
kube-controller-manager	Liveness
kube-proxy	Liveness
kube-scheduler	Liveness
pod-checkpointer	None	Does not have any way to probe at the moment.

See also:

• https://srcco.de/posts/kubernetes-liveness-probes-are-dangerous.html
• kubernetes-retired/bootkube#495

Constant diff in `lokoctl cluster apply`

When re-running lokoctl cluster install on a cluster which was just deployed and should therefore be aligned with the desired configuration, we see a constant diff:

Terraform will perform the following actions:

  # module.packet-johannes-test.module.bootkube.template_dir.bootstrap-manifests will be created
  + resource "template_dir" "bootstrap-manifests" {
      + destination_dir = "../cluster-assets/bootstrap-manifests"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/bootstrap-manifests"
      + vars            = {
          + "cloud_provider"    = ""
          + "etcd_servers"      = "https://johannes-test-etcd0.dev.lokomotive-k8s.net.:2379"
          + "hyperkube_image"   = "k8s.gcr.io/hyperkube:v1.17.3"
          + "pod_cidr"          = "10.2.0.0/16"
          + "service_cidr"      = "10.3.0.0/16"
          + "trusted_certs_dir" = "/usr/share/ca-certificates"
        }
    }

  # module.packet-johannes-test.module.bootkube.template_dir.calico[0] will be created
  + resource "template_dir" "calico" {
      + destination_dir = "../cluster-assets/charts/kube-system/calico"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/calico"
    }

  # module.packet-johannes-test.module.bootkube.template_dir.kube-apiserver will be created
  + resource "template_dir" "kube-apiserver" {
      + destination_dir = "../cluster-assets/charts/kube-system/kube-apiserver"
      + id              = (known after apply)
      + source_dir      = "../lokomotive-kubernetes/bootkube/resources/charts/kube-apiserver"
    }
...

Confirming the change results in a successful terraform apply which doesn't seem to have any negative effects, however the diff probably shouldn't be there.

Adding/removing controller nodes to existing cluster does not work

It seems that new etcd member is not able to join the cluster.

However, if you create 3-controller node cluster from scratch and then you need to replace one one (so the data from the node is lost), everything is fine.

Remove dependency on kubelet-wrapper and etcd-wrapper from FCL

Currently we rely on kubelet-wrapper and etcd-wrapper scripts provided by Flatcar Container Linux, which may become problematic if we need some changes to those scripts etc.

We should perhaps ship our own version of scripts, which we have full control over, so we minimize the amount of dependencies between Lokomotive and Flatcar Container Linux.

During the process, we can also consider using docker instead of rkt.

Consider encrypting in-cluster traffic

Right now in-cluster traffic is unencrypted. For cases where users don't trust the network this might be problematic.

We can be solve this in different ways (using a CNI plugin with encryption support, using a service mesh...)

This issue tracks discussions on this topic.

Changing kubernetes version triggers recreation of AWS worker pools

Since we use self-hosted kubelet and this is the strategy for node updates, we should probably ignore changes to pool user_data field.

Validate cluster config

General

We should consider adding validation for cluster configuration. At the moment we catch some configuration problems (mainly wrong HCL syntax), however we probably want to have more Lokomotive-specific validation.

The rationale for this task:

• Being able to catch config problems early rather than starting to execute only to discover that we have some missing/wrong piece of config later down the line. If validation passes, the cluster should deploy (barring any run time issues).
• Having one place to validate interaction between various knobs. Configuration can grow complex over time, and various features and knobs can have unexpected interactions. Having one place to validate the config and write unit tests for all config-related aspects can make this aspect much easier.
• Being able to provide similar-looking, meaningful errors to the user.
• Less chance for missed test cases when modifying knobs.

Right now we have validation code scattered throughout the codebase. Examples:

• lokomotive/pkg/platform/packet/packet.go

  Lines 133 to 136 in 8e0529a

  	if c.AuthToken == "" && os.Getenv("PACKET_AUTH_TOKEN") == "" {
  	return fmt.Errorf("cannot find the Packet authentication token:\n" +
  	"either specify AuthToken or use the PACKET_AUTH_TOKEN environment variable")
  	}

• lokomotive/pkg/components/cluster-autoscaler/component.go

  Lines 239 to 250 in 8e0529a

  	switch c.Provider {
  	case "packet":
  	diagnostics = c.validatePacket(diagnostics)
  	default:
  	// Slice can't be constant, so just use a variable
  	supportedProviders := []string{"packet"}
  	diagnostics = append(diagnostics, &hcl.Diagnostic{
  	Severity: hcl.DiagError,
  	Summary: "Make sure to set provider to one of supported values",
  	Detail: fmt.Sprintf("provider must be one of: '%s'", strings.Join(supportedProviders[:], "', '")),
  	})
  	}

• lokomotive/pkg/dns/dns.go

  Lines 62 to 77 in 8e0529a

  	func ParseDNS(config *Config) (DNSProvider, error) {
  	// Check that only one provider is specified.
  	if config.Provider.Manual != nil && config.Provider.Route53 != nil {
  	return DNSNone, fmt.Errorf("multiple DNS providers specified")
  	}
  	if config.Provider.Manual != nil {
  	return DNSManual, nil
  	}
  	if config.Provider.Route53 != nil {
  	return DNSRoute53, nil
  	}
  	return DNSNone, fmt.Errorf("no DNS provider specified")
  	}

Implementation

Important points to consider:

• Config validation should be centralized. For example, we want to validate component configuration even if the logic for deploying components is triggered after cluster installation had been completed.

I think we can extract all config-related logic from the various packages and move them into pkg/config. There we can validate things centrally and have one place for all config-related unit tests. We can then have a nice state where any value used in platform or component code can be treated as validated, which could simplify the code and likely remove some ad-hoc validation checks. I think a good rule of thumb could be - if it has an hcl:"stuff" kind of tag, it should be in pkg/config. The other packages can contain package-specific data structure which are derived from those in the config package, but shouldn't deal with unmarshalling HCL and/or config validation.

ExternalDNS component doesn't support default AWS credentials

It only supports the env variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY but it should also support ~/.aws/credentials to make it consistent with the rest of lokoctl.

Taint bootstrap kubelet and drain the node before shutting down self-hosted kubelet

To avoid the impact of configuration drift between self-hosted and bootstrap kubelet, it make sense for bootstrap kubelet to only handle self-hosted kubelet pod. All other pods should be then handled with self-hosted kubelet.

This way, if bootstrap kubelet does not have certain option configured (e.g. host mount), it won't affect the pod needing it (for example when mounting PVCs)

We could make bootstrap kubelet register with extra taint, which then self-hosted kubelet should remove after it's ready.

The same way, on shutdown of self-hosted kubelet, we could use PreStop hook to taint and drain the node. That ensures that updating self-hosted kubelet is safe and has a small impact on the cluster.

Expose MetalLB node peer config options in lokoctl

General

MetalLB supports a set of BGP peer configuration options when configured via a ConfigMap. Our "peer autodiscover" feature allows configuring BGP peers via k8s node labels as well.

We should allow users to tweak the options supported by MetalLB using Lokomotive cluster configuration.

Following is a list of the options:

• Local ASN (discovered automatically on Packet)
• Remote ASN (always 65530 on Packet)
• Peer address (discovered automatically on Packet)
• Peer port (always 179 on Packet)
• Router ID (an empty value is handled by MetalLB)
• Hold timer (defaults to 90s in MetalLB)
• BGP password (can be retrieved from the Packet API using Terraform)

It seems the most important options to add are hold timer and BGP password, because the rest of the options are currently handled automatically one way or the other. Still, for better flexibility (and for bare metal environments other than Packet) we may want to expose all the knobs in the cluster config.

Implementation

Specifying any peer-related config options on Packet should be per worker pool.

Add Bash/zsh autocompletion

Cobra supports generating autocomplete files for Bash and zsh. We could leverage this to make working with lokoctl nicer.

Tighten firewall rules during node boot window

When a new node is created, it starts with a basic set of firewall rules before getting full dynamic firewall rules from the Kubernetes control plane.

There is a small (few minutes) window while the node starts where the basic firewall rules have been applied and the node is internet exposed more broadly than it could be. We should lock down the exposed ports to just the cluster IP range earlier in the boot process so that the exposure is reduced.

Packet: Add support for hardware reservations

It should be possible to specify hardware reservations in Packet (supported in terraform code).

However, there is an issue in packet terraform provider in which we need to create some worker pools before another when using hardware reservations "next-available" and a specific UUID. Upstream bug was reported here and the "fix" was to document to create such dependencies manually.

Without that (workaround in terraform is just doing terraform apply -target ...) is not possible to reliably create clusters that mix "next-available" and specific UUIDs.

Such clusters are useful because you may want to run with reserved instances only (for pricing reasons) but you need to spread some workers in specific ways and need to specify UUIDs too.

Document using Envoy's annotations for Contour external-dns integration

Contour component supports ingress_hosts parameter, which can be used for integrating Contour with external-dns until #13 is resolved.

We should document this workaround.

Update error wrapping to new Golang conventions

Since Go 1.13 there's a new way to work with errors. We should go through our use of fmt.Errorf and errors.Wrap* to make sure we are following it.

Deploy some components by default

Description

There are some components, which we want to ship in every installation, like metrics-server. For that, lokoctl should support list of default components, which will be deployed without any configuration.

The alternative would be to put them into bootkube, but that does not play well with customization etc.

We should also decide, if we want to have an option to disable default components, if some users don't want them.

Initial task breakdown

• Decide if we should be able to disable default components
• Define a way to mark components as default

Acceptance Criteria

• Lokoctl component can be marked as default and will be installed on default cluster installation

Add contributing guidelines and contributor docs

Provide general guidelines and docs on how to find low-hanging-fruit, easy-first issues, how to clone the repo and getting started, etc.

Remove documentation duplicates

Once we have the component configuration reference guides, the howtos, and the quickstarts we should remove any documentation duplicates.

Create better README

It was taken from the lokoctl command but it should explain more about Lokomotive in general and provide the big picture.

We can consider removing the README in assets/lokomotive-kubernetes.

Make control plane bootstrap resilient to random failures

General

In some cases, the control plane bootstrap process randomly fails due to reasons beyond our control such as a random error from a container registry or a momentary network disruption.

The following log output demonstrates such a failure.

lokoctl output:

module.packet-albanpacketcluster.null_resource.bootkube-start (remote-exec): Job for bootkube.service failed because the control process exited with error code.
module.packet-albanpacketcluster.null_resource.bootkube-start (remote-exec): See "systemctl status bootkube.service" and "journalctl -xe" for details.

Logs from the controller node:

bootkube-start[1312]: 6b69eb11d043: Waiting
bootkube-start[1312]: 0abeb1500767: Waiting
bootkube-start[1312]: error pulling image configuration: Get https://storage.googleapis.com/eu.artifacts.google-containers.appspot.com/containers/images/sha256:09d3b1c54c94860748be676d1959cc7ab667052363755c706838f0bfa75c20a1: read tcp 147.75.85.101:55542->172.217.19.208:443: read: connection reset by peer

In these cases, we should have a retry mechanism somewhere in our stack which can compensate for such an error.

Implementation

We should figure out the right place to add the retry logic.

• Does it make sense to add retry functionality to Bootkube?
• Does it make sense to add retry logic to the bootkube-start script without modifying Bootkube?

disparity in linter run locally and in CI

When I run the linter locally I see following errors:

$ make lint 
golangci-lint run --enable-all --disable=godox --max-same-issues=0 --max-issues-per-linter=0 --build-tags aws,packet,e2e,disruptive-e2e,prometheus --new-from-rev=$(git merge-base $(cat .git/resource/base_sha 2>/dev/null || echo "master") HEAD) --modules-download-mode=vendor --timeout=5m ./...
WARN [runner] Can't run linter goanalysis_metalinter: ST1008: failed prerequisites: [email protected]/kinvolk/lokomotive/pkg/backend 
WARN [runner] Can't run linter unused: buildssa: analysis skipped: errors in package: [/home/surajd/work/lokomotive/pkg/backend/backend.go:18:2: could not import fmt (/usr/local/go/src/fmt/errors.go:7:8: could not import errors (/usr/local/go/src/errors/wrap.go:8:2: could not import internal/reflectlite (/usr/local/go/src/internal/reflectlite/value.go:8:2: could not import runtime (/usr/local/go/src/runtime/alg.go:8:2: could not import internal/cpu (-: could not load export data: cannot import "internal/cpu" (unknown iexport format version 1), export data is newer version - update tool))))) /home/surajd/work/lokomotive/pkg/backend/backend.go:20:2: could not import github.com/hashicorp/hcl/v2 (/home/surajd/work/lokomotive/vendor/github.com/hashicorp/hcl/v2/diagnostic.go:4:2: could not import fmt (/usr/local/go/src/fmt/errors.go:7:8: could not import errors (/usr/local/go/src/errors/wrap.go:8:2: could not import internal/reflectlite (/usr/local/go/src/internal/reflectlite/value.go:8:2: could not import runtime (/usr/local/go/src/runtime/alg.go:8:2: could not import internal/cpu (-: could not load export data: cannot import "internal/cpu" (unknown iexport format version 1), export data is newer version - update tool)))))) /home/surajd/work/lokomotive/pkg/backend/backend.go:26:14: undeclared name: hcl /home/surajd/work/lokomotive/pkg/backend/backend.go:26:25: undeclared name: hcl /home/surajd/work/lokomotive/pkg/backend/backend.go:26:42: undeclared name: hcl /home/surajd/work/lokomotive/pkg/backend/backend.go:20:2: "github.com/hashicorp/hcl/v2" imported but not used] 
ERRO Running error: buildssa: analysis skipped: errors in package: [/home/surajd/work/lokomotive/pkg/backend/backend.go:18:2: could not import fmt (/usr/local/go/src/fmt/errors.go:7:8: could not import errors (/usr/local/go/src/errors/wrap.go:8:2: could not import internal/reflectlite (/usr/local/go/src/internal/reflectlite/value.go:8:2: could not import runtime (/usr/local/go/src/runtime/alg.go:8:2: could not import internal/cpu (-: could not load export data: cannot import "internal/cpu" (unknown iexport format version 1), export data is newer version - update tool))))) /home/surajd/work/lokomotive/pkg/backend/backend.go:20:2: could not import github.com/hashicorp/hcl/v2 (/home/surajd/work/lokomotive/vendor/github.com/hashicorp/hcl/v2/diagnostic.go:4:2: could not import fmt (/usr/local/go/src/fmt/errors.go:7:8: could not import errors (/usr/local/go/src/errors/wrap.go:8:2: could not import internal/reflectlite (/usr/local/go/src/internal/reflectlite/value.go:8:2: could not import runtime (/usr/local/go/src/runtime/alg.go:8:2: could not import internal/cpu (-: could not load export data: cannot import "internal/cpu" (unknown iexport format version 1), export data is newer version - update tool)))))) /home/surajd/work/lokomotive/pkg/backend/backend.go:26:14: undeclared name: hcl /home/surajd/work/lokomotive/pkg/backend/backend.go:26:25: undeclared name: hcl /home/surajd/work/lokomotive/pkg/backend/backend.go:26:42: undeclared name: hcl /home/surajd/work/lokomotive/pkg/backend/backend.go:20:2: "github.com/hashicorp/hcl/v2" imported but not used] 
make: *** [Makefile:50: lint] Error 3

but when the same is run in the CI I can see following output:

Running linter (if available)
golangci-lint run --enable-all --disable=godox --max-same-issues=0 --max-issues-per-linter=0 --build-tags aws,packet,e2e,disruptive-e2e,prometheus --new-from-rev=$(git merge-base $(cat .git/resource/base_sha 2>/dev/null || echo "master") HEAD) --modules-download-mode=vendor --timeout=5m ./...
test/monitoring/control-plane_test.go:86:26: Using the variable on range scope `tc` in function literal (scopelint)
			t.Logf("querying %q", tc.query)
			                      ^
test/monitoring/control-plane_test.go:92:47: Using the variable on range scope `tc` in function literal (scopelint)
			results, warnings, err := v1api.Query(ctx, tc.query, time.Now())
			                                           ^
test/monitoring/control-plane_test.go:100:82: Using the variable on range scope `tc` in function literal (scopelint)
			t.Logf("found %d results for %s", len(strings.Split(results.String(), "\n")), tc.query)
			                                                                              ^
test/monitoring/control-plane_test.go:33: Function 'TestPrometheusMetrics' is too long (69 > 60) (funlen)
func TestPrometheusMetrics(t *testing.T) {
test/components/util/util.go:251:2: assignments should only be cuddled with other assignments (wsl)
	p.portForwarder = forwarder
	^
test/components/util/util.go:257:3: expressions should not be cuddled with blocks (wsl)
		t.Logf("output of port forwarder:\n%s\n", out.String())
		^

Kubelet is not allowed to use privileged PSP

And because of this, following error is printed:

E0226 10:39:09.237346   12034 kubelet.go:1662] Failed creating a mirror pod for "kube-apiserver-6f5b85f5db-6n2q8-ip-10-0-2-169_kube-system(8122a931b691267d810b16f3f2435a8d)": pods "kube-apiserver-6f5b85f5db-6n2q8-ip-10-0-2-169" is forbidden: unable to validate against any pod security policy: [spec.volumes[1].hostPath.pathPrefix: Invalid value: "/usr/share/ca-certificates": is not allowed to be used spec.volumes[2]: Invalid value: "emptyDir": emptyDir volumes are not allowed to be used spec.securityContext.hostNetwork: Invalid value: true: Host network is not allowed to be used spec.volumes[0]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used spec.volumes[3]: Invalid value: "hostPath": hostPath volumes are not allowed to be used]

We already allow kubelet to use pod-checkpointer PSP here: https://github.com/kinvolk/lokomotive/blob/master/assets/charts/control-plane/kubernetes/templates/kubelet-pod-checkpointer-psp-role-binding.yaml. Perhaps we should do the same for privileged policy (or whatever PSP is used for kube-apiserver.

Add component node problem detector

Add this as part of the prometheus operator(monitoring) component or as a separate component.

https://github.com/kubernetes/node-problem-detector

Reconsider running metallb controllers on controller nodes

In #69 we changed the metallb component to run speakers on worker nodes only and to run controllers on controller nodes to avoid "unnecessary interruptions to MetalLB due to cluster scaling events".

Maybe there's a way to solve both problems without running controllers on controllers (:D), we should investigate that.

SIGINT to lokoctl is not handled gracefully

Right now receiving SIGINT in lokoctl (Ctrl-C) is not handled so if, say, this happens while terraform is running we might end up in a bad state.

We should handle SIGINT and exit gracefully.

Document Pod Security Policies

Description

Document the PSP setup and how users can/should extend the existing policies for "optimal security".

Initial task breakdown

• Document default PSPs (privileged and restricted)
• Give recommendations for creation and usage of PSPs

Acceptance Criteria

• Documentation for the above tasks is deployed on the Lokomotive docs site

Rename PSP policies alphabetically

The PSP policy order is explained here: https://kubernetes.io/docs/concepts/policy/pod-security-policy/#policy-order

When adding a new PSP in this PR: https://github.com/kinvolk/terraform-render-bootkube/pull/15/files we hit issues were different policies were used because of the alphabetical order of the PSP names.

Therefore, the name was chosen to be first in alphabetical ordern, but is not quite clear for an Engineer and is very error prone.

We can use a format that starts with a number, like 01-xx, 02-yy, and be sure PSP order is explicit to the engineer writing them.

Writing assets to disk sometimes produces bad Terraform code

I'm not sure if this is only the issue when doing local development or also with standalone binaries, but I sometimes get some corrupted lines in assets files. Example:

$ cat lokomotive-assets/lokomotive-kubernetes/packet/flatcar-linux/kubernetes/bootkube.tf 
module "bootkube" {
  source       = "../../../bootkube"
  cluster_name = var.cluster_name

  # Cannot use cyclic dependencies on controllers or their DNS records
  api_servers          = [local.api_fqdn]
  api_servers_external = [local.api_external_fqdn]
  etcd_servers         = local.etcd_fqdn
  asset_dir            = var.asset_dir
  networking           = var.networking
  network_mtu          = var.network_mtu

  # Select private Packet NIC by using the can-reach Calico autodetection option with the first
  # host in our private CIDR.
  network_ip_autodetection_method = "can-reach=${cidrhost(var.node_private_cidr, 1)}"

  pod_cidr              = var.pod_cidr
  service_cidr          = var.service_cidr
  cluster_domain_suffix = var.cluster_domain_suffix
  enable_reporting      = var.enable_reporting
  enable_aggregation    = var.enable_aggregation

  certs_validity_period_hours = var.certs_validity_period_hours

  container_arch = var.os_arch
}
 expose_on_all_ports = true
}

Makefile: add 'go test -run=nonexistent -tags e2e,....' to 'build' target

So also all e2e tests are compiled locally. That allows to catch syntax errors earlier and should prevent CI from creating cluster with broken e2e tests.

CC @iaguis

Ingress component does not work with external-dns to create DNS entries based on Ingress objects

Currently as Ingress Controller implementation, we use Contour, which at the moment does not set the IP value in Ingress resources, which makes it unusable with external-dns component.

Example:

$ kubectl get ingress --all-namespaces
NAMESPACE   NAME      HOSTS                                    ADDRESS   PORTS     AGE
dex         dex       dex.london.lokomotive-k8s.net                 80, 443   23h
gangway     gangway   gangway.london.lokomotive-k8s.net             80, 443   23h

This has been a long standing issue for Contour (projectcontour/contour#403) from 2018.

Workarounds

• use Envoy annotations and configure external-dns to also create entries based on Service objects annotations
• Create alternative implementation for Ingress component using e.g. nginx-ingress and recommend using that if user needs external-dns integration

Actual solution

We could join efforts in projectcontour/contour#2121 and implement this functionality in Contour.

'make docs' updates unchanged files

Perhaps we could avoid including the date footer in generated docs, as if one runs make docs on the day other than the docs were generated, all files then include useless diff:

diff --git a/docs/cli/lokoctl.md b/docs/cli/lokoctl.md
index b1b5804b..9937a0f2 100644
--- a/docs/cli/lokoctl.md
+++ b/docs/cli/lokoctl.md
@@ -22,4 +22,4 @@ Manage Lokomotive clusters.
 * [lokoctl health](lokoctl_health.md)   - Get the health of a Lokomotive cluster
 * [lokoctl version](lokoctl_version.md)         - Show version
 
-###### Auto generated by spf13/cobra on 26-Feb-2020
+###### Auto generated by spf13/cobra on 28-Feb-2020
diff --git a/docs/cli/lokoctl_cluster.md b/docs/cli/lokoctl_cluster.md
index 75aa7c2a..5676de2a 100644
--- a/docs/cli/lokoctl_cluster.md
+++ b/docs/cli/lokoctl_cluster.md
@@ -26,4 +26,4 @@ Manage a Lokomotive cluster
 * [lokoctl cluster destroy](lokoctl_cluster_destroy.md)         - Destroy Lokomotive cluster
 * [lokoctl cluster install](lokoctl_cluster_install.md)         - Install Lokomotive cluster with components
 
-###### Auto generated by spf13/cobra on 26-Feb-2020
+###### Auto generated by spf13/cobra on 28-Feb-2020
diff --git a/docs/cli/lokoctl_cluster_destroy.md b/docs/cli/lokoctl_cluster_destroy.md
index 14f088ac..c5d8b139 100644
--- a/docs/cli/lokoctl_cluster_destroy.md
+++ b/docs/cli/lokoctl_cluster_destroy.md
@@ -30,4 +30,4 @@ lokoctl cluster destroy [flags]
 
 * [lokoctl cluster](lokoctl_cluster.md)         - Manage a Lokomotive cluster
 
-###### Auto generated by spf13/cobra on 26-Feb-2020
+###### Auto generated by spf13/cobra on 28-Feb-2020
diff --git a/docs/cli/lokoctl_cluster_install.md b/docs/cli/lokoctl_cluster_install.md
index aee96762..5cb54a39 100644
--- a/docs/cli/lokoctl_cluster_install.md
+++ b/docs/cli/lokoctl_cluster_install.md
@@ -31,4 +31,4 @@ lokoctl cluster install [flags]
 
 * [lokoctl cluster](lokoctl_cluster.md)         - Manage a Lokomotive cluster

Ideally, we would not remove the date from the footer, but only update it, if there are some other updates to the file.

Run upgrade tests from master to PR code as part of CI process

To ensure, that our users can update to the next version of Lokomotive, we should test upgrade process via CI, the same way we test fresh installations.

Monitor all the serviceMonitors in the cluster

Prometheus-operator is currently not monitoring all the ServiceMonitors. So sets the value of serviceMonitorSelectorNilUsesHelmValues to false to make sure that all ServiceMonitors are selected.

When the value of serviceMonitorSelectorNilUsesHelmValues is true the Prometheus object's field serviceMonitorSelector is set with following values:

  serviceMonitorSelector:
    matchLabels:
      release: "prometheus-operator"

Due to this any ServiceMonitor object that does not have above labels, they won't be selected. For Lokomotive we would want to keep track of all ServiceMonitors in the cluster.

And when serviceMonitorSelectorNilUsesHelmValues is set to false, serviceMonitorSelector field in Prometheus object is empty.

Provide super-power kubeconfig for emergency scenarios

Currently, Terraform code creates a kubeconfig you can use to access the cluster without any restriction (RBAC, etc.).

It is useful for some users to have such a file in emergency scenarios, where for some reason or another Dex/Gangway are failing and need to recover the cluster.

Expose read-only CI results to everyone

We want to have a read-only view of the CI results exposed to the public so they can see the results of the CI test.

Cut release v0.1.0

Once we get everything we want merged we should cut the first Lokomotive release.

packet: expose disabling BGP to lokoctl

In the terraform layer we have a disable_bgp parameter, we should expose this in lokoctl so it's accessible to users.

Document release process

We should document how we do a release.

This can be for now a manual list of steps to produce the release artifact and release notes and we can automate later.

We can generate this as we cut the first release.

Add lokoctl command to uninstall a component.

Piping the output of lokoctl render-manifest <component> to kubectl delete -f or using helm to manage the components after installation is not the recommend way of managing components.

Fix MetalLB speaker and controller scheduling

At the moment we have two problems with the way we schedule MetalLB pods:

1. Speaker pods are scheduled on both worker nodes and controller nodes.
2. The controller pod is scheduled on a worker node.

We should ensure speaker pods never run on controller nodes because controller nodes don't serve application traffic and therefore don't need to participate in BGP. Currently we set the following toleration for speaker pods:

lokomotive/pkg/components/metallb/component.go

Lines 70 to 73 in df30e7d

	c.SpeakerTolerations = append(c.SpeakerTolerations, util.Toleration{
	Effect: "NoSchedule",
	Key: "node-role.kubernetes.io/master",
	})

We should ensure the controller pod runs on a controller node to avoid unnecessary interruptions to MetalLB due to cluster scaling events.

Add CI check for copyright headers

All files should be annotated with the license.

Address concerns for the DNS abstraction

We added DNS abstraction to support different DNS providers instead of just Route53. However, there were some concerns about it not being generic enough because it's not a portable Terraform target.

This issue tracks those concerns.

enable monitoring of etcd in the default prometheus-operator component

For etcd you need to provide necessary certificates to the prometheus-operator to ensure that it can scrape metrics from these components.

etcd

For etcd add following snippet to the values file.

kubeEtcd:
  enabled: true
  endpoints: {{.EtcdEndpoints}}
  service:
    port: 2381
    targetPort: 2381
  serviceMonitor:
    scheme: https
    insecureSkipVerify: false
    caFile: /etc/prometheus/secrets/etcd-client-cert/etcd-ca
    certFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client
    keyFile: /etc/prometheus/secrets/etcd-client-cert/etcd-client-key

But to ensure this works create a secret named etcd-client-cert in the same namespace as prometheus operator and it should have three keys etcd-ca, etcd-client, etcd-client-key.

Right now we ask user to mention the etcd endpoints but this information is already available in terraform state and it can be retrieved from there and fed to component information internally.

Also etcd-client certs are generated on cluster install, using those certs for prometheus is not ideal because they have wide pemissions. So creating a user with limited permissions to read metrics and then creating certs for that user is the ideal way to go forward.

Fix calico-related bootkube errors

When booting a cluster, the following error is shown in the bootkube logs:

Sep 11 12:54:02 kinvolk-dev-test-controller-0 bootkube-start[1952]: [ 1532.151750] bootkube[6]: Failed creating /assets/manifests/calico-policy.yaml  : dicovery failed: discover group verison : groupVersion shouldn't be empty

This doesn't seem to cause any harm but it should not happen.

packet: print error/warning when os_version is specified and not ipxe_script_url

os_version only works for IPXE, print an error when this parameter is specified and ipxe_script_url is not. Currently there is not any error/warning, the installation continues installing a different version.

Consider using Code Climate and Codecov for checking code quality and coverage

From my experience, those 2 tools integrates very well with GitHub workflow and are free for open source projects. Code Climate work great for catching unnecessary complexity, duplicated code etc and Codecov makes browsing code coverage very easy via browser.