kahkhang/kube-linode

:whale: Provision a Kubernetes/CoreOS cluster on Linode

kube-linode's Issues

Support for private networks

This is a lower-priority item for me, personally, but it would be great if this supported private networking to avoid possible bandwidth charges.

Allow custom usernames

Worker not deployed

Master was successfully span up, but not Workers, it stuck at Brand New status somehow. See:

Add ``bc`` to dependencies

When attempting to use the script on a fresh install of Ubuntu 16.04, I received the following error after inputting my API key:

? Select a datacenter (Use arrow keys)
/root/.kube-linode/inquirer_common.sh: line 88: bc: command not found
/root/.kube-linode/inquirer_common.sh: line 88: bc: command not found
/root/.kube-linode/inquirer_common.sh: line 88: bc: command not found

Migrate API from v3 to v4

Try out OpenEBS

Rook is awesome but is a little too memory intensive. Perhaps OpenEBS would offer a more lightweight solution since it's built from the ground up.

FAQ?

This project looks neat and I'd like to try it out, but I have a few questions before I give it an API key. I figured it might be useful to start an issue for questions like this.

Can this be run on an existing server, or will start its own?

Does this require multiple servers? The .gif example shows two.

Is there a minimum size recommendation for the server? Will this work on a single $5 linode, or does it require more resources?

What actions can this program perform on my Linode account? It's probably a good idea to have this in the docs, if the program can do anything to an account that would incur fees.

Edit: Sorry, I wrote this on mobile and hit "submit" before I meant to.

Storage:ceph:rook. No nodes are available that match all of the following predicates:: MatchNodeSelector (1), PodToleratesNodeTaints (1).

After my 2 node cluster has been running for a day or so I notice a number of the pods in the rook namespace have the following status.

No nodes are available that match all of the following predicates:: MatchNodeSelector (1), PodToleratesNodeTaints (1).

Pods are in pending state.

What debugging information can I supply that is more pertinent to resolve issue?
What is the next move to get back to healthy state?

Validate SSH host keys

A number of ssh commands specify -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no. It would be far better to retrieve the host's SSH keys from the provider (not sure if that's possible with Linode's API, Digital Ocean and AWS both allow access to the system's startup messages, which include the host's public SSH keys)

Rook operator fails to run because POD_NAMESPACE is not provided

2017-10-22 21:01:33.216728 I | rook: starting Rook v0.5.0-195.g84593aa with arguments '/usr/local/bin/rook operator'
2017-10-22 21:01:33.216863 I | rook: flag values: --help=false, --log-level=INFO, --mon-healthcheck-interval=20s, --mon-out-timeout=5m0s
2017-10-22 21:01:33.221422 I | rook: starting operator
failed to run operator. Rook operator namespace is not provided. Expose it via downward API in the rook operator manifest file using environment variable POD_NAMESPACE

Customize storage disk size

When building a cluster using the smallest available Linode, a storage disk size of 10240MB is great. However, if you're using a larger Linode, only having 10GB available for local storage is a bit of a waste. As far as I can see with a quick grep, it doesn't look like STORAGE_DISK_SIZE is customizable right now.

Perhaps there could be an option to set it explicitly, or do a percentage of total disk size per Linode plan, or
do a fixed size for the CoreOS disk and leave the remainder for local storage.

Add Rook Grafana/Prometheus Monitoring

Exposing a Service via Traefic

I have a custom app that I have deployed on my kube-linode instance.
Is there any recommendation about how best to add a k8s service to the project instance of Traefic such that it will support current and future project releases?

Clairify what email is used for

The setup process asks the user for an email address, but does not explain how it will be used. As far as I can tell it's used for the ACME registration, but it would be nice to see that in the prompt, maybe even stating if it's okay to not provide an email/how that email will be used.

Possible to get bare bones version?

Real Nice project - would be great to have with this just a bare cluster ( no traefik, rook, prometheus etc )
Cheers,

Quick question (DNS)

Hi, This looks amazing and I'm hoping I can use it.
I just had one question and that was, is it required to use Linode's DNS with this? My team uses Route53 for our DNS management currently and I can't switch that over to Linode.

Time to provision?

Hi, this project looks fabulous. Thank you for putting it together!

I tried to use it a couple times today, but I feel like it's a bit stuck. Do you have an estimate for about how long it should take to provision each linode? I let it run twice, each time for over an hour, with no end in sight simply setting up the master node.

I'll leave it running a bit longer this time just to see if I'm too impatient.

SELinux: mount invalid. Same superblock, different security settings for (dev mqueue, type mqueue)

Hi,
I have run the script a couple of times against SINGAPORE and CA Datacentres and it continues to produce the error in the log snippet below whilst boot looping. Please advise.

My settings.env is as follows:
DATACENTER_ID=3
MASTER_PLAN=2
WORKER_PLAN=1
NO_OF_WORKERS=1

Log snippet

localhost login: [   56.033565] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready

This is localhost (Linux x86_64 4.12.7-coreos) 06:49:43

eth0: 45.33.45.126 2600:3c01::f03c:91ff:fe92:f7b

localhost login: [   62.799469] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   62.890600] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   62.906354] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)
[   62.922932] SELinux: mount invalid.  Same superblock, different security settings for (dev mqueue, type mqueue)

Document DNS record destruction

When I first ran kube-linode, I was a bit surprised to see that it destroyed DNS records that were previously configured for Google Apps for my domain. It would be helpful to point this out in the README, perhaps also with a prompt at runtime that confirms the user is ok with removing any existing DNS records.

Setup integration tests

Spin up one master/worker node, perform some kubectl verification to see if the nodes/pods are started correctly

Document acme.json rate limit / use of staging server

Increase Rook metrics scrapeTimeout

Private networking question

Is k8s internal traffic using private ipv6 networking? I was originally only looking at ipv4 traffic and thought all k8s internal traffic was running over the public interface but now I'm wondering if it's using ipv6. I don't see any ipv6 config in the manifests though 🤔

Either way, as long as the internal traffic is over the private network and is protected by (at least) iptables then I'm satisfied

Passing in arguments to ./kube-linode?

Hey - I'm working on a personal prototyping pipeline and I'd love to be able to call your scripts from within my own scripts. How would you prefer this be done? Would be happy to make a PR I just wanted to check with you first.

To be clear, it would be awesome to be able to do something like the following:

./kube-linode.sh --API_KEY=$(api_key) --WORKERS=2

etc.

Invalid Domain on subdomain

I use CloudFlare as my primary DNS host, and delegate a subdomain to linode, cloud.example.com.

When trying to use the domain on kube-linode, it requires a TLD, although I have the subdomain added into linode.

Verify contents of downloaded files

I noticed that install-coreos.sh downloads ct (and coreos-install) from github, explicitly disabled certificate validation, and runs it as root. Certificate validation being disabled is questionable at best, but regardless the GPG signature (which CoreOS publishes) of the ct binary should be validated, or include the hash of the binary in the script and validate against that (easier to implement, but slightly harder to change the ct version). coreos-install.sh also downloads a shell script from github, with certificate validation enabled this time, but still does not validate it against anything.

Add timestamp to ~/.kube/config.bak

Make email address optional

ACME registration doesn't require an email, so the user shouldn't be force to provide one. Allowing them to enter one is fine, but maybe say "(optional)" after the prompt. Not sure if traefik will allow this, but I dont see any reason why it wouldn't.

Add Elasticsearch/Fluentd/Kibana stack

Rook Agent Pods fail to start

See #51

Error: failed to start container "rook-agent": Error response from daemon:
{"message":"mkdir /usr/libexec/kubernetes: read-only file system"}
Error syncing pod
Back-off restarting failed container

My hunch is that some flag needs to be added to the kubelet.service systemd file (which can be found at https://github.com/kahkhang/kube-linode/blob/master/manifests/container-linux-config.yaml and https://github.com/kahkhang/kube-linode/blob/master/manifests/container-linux-config-worker.yaml) to include --volume-plugin-dir=/etc/kubernetes/volumeplugins, and also probably the kubernetes version needs some bumping up as well to the latest one which supports flex volume plugins.

Add verbose logging

HA enhancements

It looks like there's at least 2 more areas of improvement needed for kube-linode,

• Multiple Masters
• NodeBalancer + Traefik NodePort service

Is this something that should wait until Terraform provisioning is in progress?
Are these already possible?

ERR_SSL_PROTOCOL_ERROR

I have previously successfully provisioned a cluster using this script. But after a teardown I ran the same script using the same domain and now I cannot access any web ui or any traefik routes because I am getting an ERR_SSL_PROTOCOL_ERROR error from the browser. I feel like I have tried everything but I can't get past this error.

Please help!

ps - I would also like to see an example of a deployment using the same subdomain and ssl...and possible an example of using this cluster but a different domain. I apologize I am not a sysadmin and ops engineer.

Use Terraform for provisioning cluster

Teardown?

Hey - working on a personal development pipeline and I have this E2E test script that tears everything down at the end. Would love to do the same for your script - would anything need to be cleaned up apart from the master/worker linode instances?

HA Setup Options

Support standalone etcd cluster, and more than one master node

Stuck on Enter Linode API Key - Urgent Help Needed

I am very excited about using this. I have been using kops on AWS and would much rather use Linode. I followed the instructions on the README

git clone https://github.com/kahkhang/kube-linode
cd kube-linode
chmod +x kube-linode.sh
./kube-linode.sh
? Enter Linode API Key (https://manager.linode.com/profile/api) :  xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
? Enter Linode API Key (https://manager.linode.com/profile/api) :  

I have verified that the API Key is good but the script just keeps asking for it. I tried just adding it to the settings.env like so:

API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

I've tries creating multiple API Keys and everyone does the same thing. I am sure this is a user error because I don't see anyone else with this problem. Please let me know what I can do to get this working.

Unfortunately, this is a rather urgent need, thanks!

Add Grafana/Prometheus support

Prevent timeout of kubectl

SSL_ERROR_INTERNAL_ERROR_ALERT on Ingress services

I just discovered this project, very cool and I'm excited to do use it. I ran it, and according to the terminal output of the script it worked, but it doesn't seem to be quite fully functional. I can use kubectl from my local machine to interact with the cluster and I can see the dashboard with kubectl proxy, but https://kube.example.com doesn't load. Firefox says:

An error occurred during a connection to kube.k8s.janky.solutions. Peer reports it experienced an internal error. Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

This seems to be the case for all of the services that are listed in the README.

The root domain and seemingly any subdomain not listed in the README, I get an invalid, self-signed cert for 8629e935e621618727d9710d574650c6.ec78bb20354254b7355bc5f0e895c417.traefik.default.

Possibly related, possibly unrelated: I noticed when browsing around in the dashboard that the rook-agent pods didn't come up, saying:

Nameserver limits were exceeded, some nameservers have been omitted, the applied nameserver line is: 173.255.243.5 173.255.244.5 173.230.145.5
Error: failed to start container "rook-agent": Error response from daemon: error while creating mount source path '/usr/libexec/kubernetes/kubelet-plugins/volume/exec': mkdir /usr/libexec/kubernetes: read-only file system
Back-off restarting failed container

And one of the rook-ceph-osd pods failed to come up, saying:

network is not ready: [runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized]
Back-off restarting failed container

I'm not really sure what is inter-related, or how to proceed. I'm happy to post any logs that are needed.

No DNS entries for kube.domain.com or traefik.domain.com

Per the readme, I expected to find DNS entries for both kube.domain.com and traefik.domain.com, but it would seem that is not actually the case.

Customization options

Hi @kahkhang thanks for this repo. It worked flawlessly 👍 I would like to know tips for changing coreos version and customize others stacks

Upgrade K8S Dashboard

Need to figure out authentication using auth header forwarding: https://github.com/kubernetes/dashboard/wiki/Access-control#authorization-header

alertmanager host not found

GIVEN I am an end-user of kube-linode
WHEN when I put the ingress monitoring host alertmanager into a web browser
THEN its should resolve to the authentication mechinism/initial welcome page.

Allow user to specify SSH key(s) to use

The current behavior appears to be to check for the presence of ~/.ssh/id_rsa and if it doesn't exist generate it, then set that as the only key that can access the CoreOS boxes. I would prefer to use a different key, so perhaps there could be a command line flag?

Also on this note, I had a single SSH key on my machine when I tried kube-linode, but it was ignored because it wasn't and RSA key. Perhaps the mechanism to detect existing keys could use something like ssh-add -L, which will include all keys.

utilities.sh issue

utilities.sh: line 116: [: =: unary operator expected

Migrate etcd2 to etcd3

Add Traefik monitoring in Grafana

Support parallel provisioning of workers

Chmod 600 /etc/traefik/acme/acme.json on master reboot

The file's permission somehow gets set to 755 upon reboot, which crashes the traefik pod.