seanknox/aztack

Terraform module for creating Kubernetes clusters running on Azure

aztack

Provision a Kubernetes cluster with Packer and Terraform on Azure Resource Manager. Inspired by Kelsey Hightower's kubestack and the tack project.

Creates a Kubernetes cluster on Azure with

• 3x Controllers
• 3x etcd
• 1x node (by default)

Software	Version
Ubuntu	16.04
cri-containerd	1.0.0.0
containerd	v1.0.0-6
runc	1.0.0-rc4+dev
etcd	v3.3.7
Kubernetes	v1.10.4
Calico	v3.1.3

More details of the cluster specs can be found in STATUS.md

Terraform

Terraform is used to declare and provision a Kubernetes cluster. Terraform runs entirely in a Docker container. The following generates Azure credentials and other required configuration and builds infra on Terraform.

$ CLUSTER_NAME=<NAME OF CLUSTER> make build post-terraform
terraform get
- module.rg
- module.vnet
- module.dns
- module.storage_account
- module.image
- module.load_balancer
- module.bastion
terraform init
Initializing modules...
- module.rg
- module.vnet
- module.dns
- module.storage_account
- module.image
- module.load_balancer
- module.bastion

Initializing provider plugins...

Resize the number of worker nodes

Edit terraform/build/$(CLUSTER_NAME)/terraform.tfvars. Set node_count to the desired value, e.g.

node_count = 5

Apply the changes:

$ CLUSTER_NAME=<NAME OF CLUSTER> make apply

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: terraform.tfstate

Packer Images

We use Packer to create an immutable image based on a source image based on Ubuntu 16.04. A prebuilt image is already provided for you, but if you wish to build yourself or change the image:

Create resource group

During the build process, Packer creates temporary Azure resources as it builds the source VM. To capture that source VM for use as an image, you must define a resource group. The output from the Packer build process is stored in this resource group.

Initialize settings

$ cd packer
$ make init

storage name: aztack1528763526
{
  "subscription_id":  "e766d9ee-d3d9-4b63-a912-8963dcfdf655",
  "client_id": "...",
  "client_secret": "...",
  "tenant_id":      "72f988bf-86f1-41af-91ab-2d7cd011db47",
  "resource_group_name": "ACStackImages",
  "location": "West US 2",
  "storage_account_name": "aztack1528763526",
  "vm_size": "Standard_D2_v2"
}

Build VHD image

$ make build
Build 'azure-arm' finished.

==> Builds finished. The artifacts of successful builds are:
--> azure-arm: Azure.ResourceManagement.VMImage:

StorageAccountLocation: westus2
OSDiskUri: https://aztack1528763526.blob.core.windows.net/system/Microsoft.Compute/Images/aztack-vhds/aztack-1528763664-osDisk.1f8be1f6-22ad-4b18-b3b3-3fe27dcfada0.vhd
OSDiskUriReadOnlySas: https://aztack1528763526.blob.core.windows.net/system/Microsoft.Compute/Images/aztack-vhds/aztack-1528763664-osDisk.1f8be1f6-22ad-4b18-b3b3-3fe27dcfada0.vhd?se=2018-07-12T00%3A46%3A43Z&sig=oSl%2BNkAEl%2FYEENeIy1Ckd9%2FgAqdAtV%2FktrdbHx3bXJ8%3D&sp=r&spr=https%2Chttp&sr=b&sv=2016-05-31
TemplateUri: https://aztack1528763526.blob.core.windows.net/system/Microsoft.Compute/Images/aztack-vhds/aztack-1528763664-vmTemplate.1f8be1f6-22ad-4b18-b3b3-3fe27dcfada0.json
TemplateUriReadOnlySas: https://aztack1528763526.blob.core.windows.net/system/Microsoft.Compute/Images/aztack-vhds/aztack-1528763664-vmTemplate.1f8be1f6-22ad-4b18-b3b3-3fe27dcfada0.json?se=2018-07-12T00%3A46%3A43Z&sig=ctdIO2s0GvBA9cA7zt6OAjQU9OY4YuVKBZIpf%2BhK0%2Bg%3D&sp=r&spr=https%2Chttp&sr=b&sv=2016-05-31

aztack's Issues

Failing "make build"

Hi,

I am just trying to get "make build" running. There is an error with cfssl. How to avoid this?

docker build -t aztack .
Sending build context to Docker daemon  637.4kB
Step 1/12 : FROM golang:1.9
 ---> ef89ef5c42a9
Step 2/12 : RUN go get -u github.com/dgrijalva/jwt-go/cmd/jwt
 ---> Using cache
 ---> 0a0a6da8c447
Step 3/12 : RUN go get -u github.com/cloudflare/cfssl/cmd/...
 ---> Running in e829a2adc0f7
# github.com/cloudflare/cfssl/csr
src/github.com/cloudflare/cfssl/csr/csr.go:272:26: cert.URIs undefined (type *x509.Certificate has no field or method URIs)
src/github.com/cloudflare/cfssl/csr/csr.go:387:7: tpl.URIs undefined (type x509.CertificateRequest has no field or method URIs)

The command '/bin/sh -c go get -u github.com/cloudflare/cfssl/cmd/...' returned a non-zero code: 2
make: *** [prereqs] Error 2

Services of type LoadBalancer not working

Creation errors with:

Events:
  Type     Reason                      Age   From                Message
  ----     ------                      ----  ----                -------
  Normal   EnsuringLoadBalancer        11s   service-controller  Ensuring load balancer
  Warning  CreatingLoadBalancerFailed  4s    service-controller  Error creating load balancer (will retry): failed to ensure load balancer for service default/nginx: [ensure(default/nginx): backendPoolID(/subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics/backendAddressPools/containerd2nics) - failed to ensure host in pool: "network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code=\"NetworkInterfaceUsesMultipleLoadBalancersOfSameType\" Message=\"Network interface /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/networkInterfaces/controller2 references more than one load balancer of the same type (internal or public): /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics-public, /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics. Only one internal and one public load balancer are allowed per availability set.\" Details=[]", ensure(default/nginx): backendPoolID(/subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics/backendAddressPools/containerd2nics) - failed to ensure host in pool: "network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code=\"NetworkInterfaceUsesMultipleLoadBalancersOfSameType\" Message=\"Network interface /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/networkInterfaces/controller3 references more than one load balancer of the same type (internal or public): /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics-public, /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics. Only one internal and one public load balancer are allowed per availability set.\" Details=[]", ensure(default/nginx): backendPoolID(/subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics/backendAddressPools/containerd2nics) - failed to ensure host in pool: "network.InterfacesClient#CreateOrUpdate: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code=\"NetworkInterfaceUsesMultipleLoadBalancersOfSameType\" Message=\"Network interface /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/networkInterfaces/controller1 references more than one load balancer of the same type (internal or public): /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics-public, /subscriptions/e766d9ee-d3d9-4b63-a912-8963dcfdf655/resourceGroups/containerd2nics/providers/Microsoft.Network/loadBalancers/containerd2nics. Only one internal and one public load balancer are allowed per availability set.\" Details=[]"]

Pods can't reach outbound internet

Pods can hit other pods, but cannot reach hosts on the internet. Nodes and controllers can, however.

May be related to #2

runc: Pods can't reach cluster service IPs

Using runc, pods cannot reach cluster service IPs (10.0.0.0/16), including apiserver at 10.0.0.1. Nodes can reach service IPs, however.

May be related to #1

To repro:

• Try curl'ing the apiserver from a pod (doesn't work):

$ kubectl run -it --image ianneub/network-tools nettools bash

If you don't see a command prompt, try pressing enter.

root@nettools-69f59c45fc-ghzmf:/# curl --connect-timeout 10 -k https://10.0.0.1
curl: (28) Connection timed out after 10000 milliseconds

curl'ing from a node does work, however:

CLUSTER_NAME=containerd2nics make ssh hostname=node1
Agent pid 56185
Identity added: .keypair/containerd2nics/containerd2nics.pem (.keypair/containerd2nics/containerd2nics.pem)
Welcome to Ubuntu 17.10 (GNU/Linux 4.13.0-39-generic x86_64)
...
ubuntu@node1:~$ curl -k https://10.0.0.1
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401