scottslowe/learning-tools

A collection of tools and files for learning new technologies

Learning Tools

This repository contains a variety of files and tools for learning new technologies. It is aimed at existing IT professionals who need some help coming up to speed with new technologies, products, or open source projects.

If you're interested in contributing, please see the guidelines.

Contents

The contents of this repository are organized according to primary technology area. Here's a quick breakdown of the major technology areas so far:

• Ansible
• CentOS Atomic Host
• Consul distributed key-value store
• CoreOS Linux
• Debian GNU/Linux
• Docker
• etcd distributed key-value store
• Fedora Linux
• Golang
• Kubernetes
• KVM
• LXC and LXD
• Network operating systems (JunOS, EOS, Cumulus, etc.)
• OpenStack
• Open vSwitch (OVS) and Open Virtual Network (OVN)
• Packer
• Photon OS
• Pulumi (primarily with Golang)
• The rkt ("Rocket") container runtime
• SSH
• Terraform
• Ubuntu Linux
• Vagrant

More information about the tools, resources, or environments for each of these technology areas can be found inside that area's README.md file.

Other Useful Projects

Here are some other projects that you may also find useful:

Kubernetes-CoreOS-Fleet-Flannel: https://github.com/kelseyhightower/kubernetes-fleet-tutorial

coreos-kubernetes-digitalocean: https://github.com/bketelsen/coreos-kubernetes-digitalocean

Docker-CoreOS-Fleet-Flannel-Etcd-Confd-Nginx: https://github.com/willrstern/production-docker-ha-architecture

If there are additional projects you feel should be added to this list, please submit a pull request. Thanks!

License

This content is licensed under the MIT License.

learning-tools's Issues

Update Consul areas to use Consul 1.0

Now that Consul 1.0 is available, all the Consul-related learning tools/environments should be updated to use the 1.0 release.

Fix intermittent provisioning errors with OVN+Docker environment

The OVN+Docker learning environment experiences intermittent errors with the Ansible playbooks.

Add area for showing how to bootstrap Ansible on Ubuntu 16.04

Since the default Ubuntu 16.04 AMIs don't come with a Python interpreter, you can't use Ansible straight "out of the box." However, it's possible to use the raw module to bootstrap Ansible. This learning environment would show how that is done (and would serve as a building block for other environments).

Can not run on VirtualBox

The box you're attempting to add doesn't support the provider
you requested. Please find an alternate box or use an alternate
provider. Double-check your requested provider to verify you didn't
simply misspell it.

If you're adding a box from HashiCorp's Atlas, make sure the box is
released.

Name: slowe/ubuntu-trusty-x64
Address: https://atlas.hashicorp.com/slowe/ubuntu-trusty-x64
Requested provider: [:virtualbox]

Add SSH key to Debian on AWS Pulumi program

The Pulumi program found in debian/debian-pulumi-aws relies on a configuration value for the SSH keypair to be used. This required configuration value can be removed by updating the program with two new resources (an SSH key and an AWS keypair).

Add learning environment for Vagrant multi-NIC networking with Ubuntu Wily

Add a learning environment for using multi-NIC networking with Vagrant and Ubuntu "Wily Werewolf". Vagrant currently has a problem with Ubuntu Wily, per this blog post:

http://blog.scottlowe.org/2016/03/21/vagrant-ubuntu-wily-networking-problem/

This learning environment will provide a workaround, either via Ansible, Jinja2 templates, ERB templates, or shell scripts.

Correct potential issues with article on Ansible and SSH bastion host

Per a reader, there are some potential issues with the article on using Ansible through an SSH bastion host (http://blog.scottlowe.org/2015/12/24/running-ansible-through-ssh-bastion-host/):

1. The use of both ProxyCommand and ForwardAgent in an SSH configuration file is not required/recommended. The reader suggests that the commands "compete" with each other.
2. Some of the SSH configurations are duplicated between the SSH configuration file and the Ansible configuration file.

I need to review these potential errors and correct them.

Intercontainer communication

Is it possible to communicate between containers on separate nodes? Can mpi applications be executed on such virtual cluster?

Add a program for using a NAT instance instead of a NAT Gateway

Options for the NAT instance include https://fck-nat.dev/stable/ (GH repo: https://github.com/AndrewGuenther/fck-nat)

[Enhancement / Suggestion] Add more resources under terraform/aws

Hey @scottslowe , loving all the content here 💯

Have been working on couple of Terraform projects and digging around to find + learn from cool resources such as yours. Was going through the Terraform code for aws at terraform/aws and figured we could possibly contribute more resources. To name a few:

1. s3
2. elasticache
3. dynamodb
4. beanstalk
5. waf 

Let me know if such contribution will add value to the existing set of learning tools, if so I'll initiate a PR :)

Regards,
Udhav

Add learning environment for using cloud-init to customize Docker

I need to add a learning environment that demonstrates using cloud-init to customize the behavior of the Docker daemon on CentOS Atomic Host.

Feature: Add Packer?

Would you mind adding packer to your folders?

Update KVM learning environment to support Libvirt provider for Vagrant

Currently the KVM learning environment (in the kvm folder) only supports the VMware provider for Vagrant (VirtualBox does not support nested virtualization). The Libvirt provider for Vagrant also supports nested virtualization, so I should update the KVM environment to also support the Libvirt provider.

Add documentation for Terraform+AWS environments

Currently, most of the Terraform+AWS environments (found in terraform/aws) have no README.md to explain how they can be used.

Error on vagrant up

Hi,

going through your instructions, I receive this error.

A Docker command executed by Vagrant didn't complete successfully!
The command run along with the output from the command is shown
below.

Command: "docker" "ps" "-a" "-q" "--no-trunc"

Stderr: Cannot connect to the Docker daemon. Is the docker daemon running on this host?

Stdout:

If I then run vagrant up again, I receive this...

A Docker command executed by Vagrant didn't complete successfully!
The command run along with the output from the command is shown
below.

Command: "docker" "run" "--name" "nginx-container" "-d" "-p" "80:80" "-p" "443:443" "nginx"

Stderr: Unable to find image 'nginx:latest' locally
latest: Pulling from library/nginx
6a5a5368e0c2: Pulling fs layer
4aceccff346f: Pulling fs layer
c8967f302193: Pulling fs layer
c8967f302193: Verifying Checksum
c8967f302193: Download complete
4aceccff346f: Verifying Checksum
4aceccff346f: Download complete
6a5a5368e0c2: Verifying Checksum
6a5a5368e0c2: Download complete
6a5a5368e0c2: Pull complete
4aceccff346f: Pull complete
c8967f302193: Pull complete
Digest: sha256:1ebfe348d131e9657872de9881fe736612b2e8e1630e0508c354acb0350a4566
Status: Downloaded newer image for nginx:latest
docker: Error response from daemon: driver failed programming external connectivity on endpoint nginx-container (8ade69b7e43443ebe5fdb52368b6cf3f9017dfb46c992cd6ed257bea095ec705): Error starting userland proxy: listen tcp 0.0.0.0:80: bind: address already in use.

Stdout: 78d3ec7da7a4031c212ea7a0fbbfed4214dadffacc0f7f6ca0842b7861bd7c42

Any insight would be helpful. I have been using Vagrant for over a year with a custom vagrant file for multiple vhosts, but am just now playing around with Docker. I am on a windows machine...I know I know haha.

Thanks!

Add CAPI+kustomize content to accompany blog post

I need to add CAPI+kustomize content to accompany a blog post I'm writing on combining CAPI with kustomize. This content will use the fictional scenario in the blog post.

Propose updates to kubeadm-vagrant instructions

Running VirtualBox as my hypervisor (on CentOS 7.5), I've found that when the K8s VMs boot up and the services are installed, they bind to the first NIC (NAT connection to the outside world) rather than the second NIC (the 192.168.100.0/24 private network). This applies to all services except the API server, and broke networking for me (all nodes use the same IP address for the first NIC, and there is no connectivity on that network between nodes anyway). It's fixable by adding the --node-ip argument to the kubelet process at startup, telling it to bind to the second NIC - e.g.

    sed -i "s/KUBELET_EXTRA_ARGS=/KUBELET_EXTRA_ARGS=--node-ip=<correct-ip-address-here>/" /etc/default/kubelet

Also, even with the config changed to use the internal network, I couldn't get Calico v3 working by following step 9 - neither the Calico nor the CoreDNS pods would fire up. However, after installing v3.3 following instructions at https://docs.projectcalico.org/v3.3/getting-started/kubernetes/, all pods run fine.

I'm presuming this is the same with other providers, but haven't tested with them. If this is an acceptable solution, I'm happy to create a pull request to update the instruction steps as per the contribution guidelines, as that would also be a useful learning experience for me!

Add a Pulumi program to deploy Flatcar Linux on Azure

As a companion to flatcar/flatcar-pulumi-aws, this Pulumi program would deploy Flatcar Linux to Azure, using a dedicated resource group, VNet, subnets, etc.

Address interface naming issues with Ubuntu "Wily" in Docker/IPVLAN environment

Some Ubuntu "Wily" 15.10 Vagrant boxes may use the ensX naming convention for network interfaces, while others will use the (older) ethX naming convention. For example, "ubuntu/wily64" uses the ethX naming style, but "slowe/ubuntu-15.10-server-amd64" uses ensX. I need to address these differences in interface naming conventions in the Docker/IPVLAN learning environment.

Improve use of Ansible roles

Of the learning environments that leverage Ansible, very few are using roles. This can make porting Ansible playbooks across learning environments more difficult. To fix this, Ansible-using learning environments need to be refactored (where possible) to use an Ansible role.

Enable upgrades for Pulumi environment role

As it is currently written, the Ansible role for provisioning a Pulumi environment won't upgrade to newer versions of Pulumi. The role needs to be fixed to enable this functionality.

Update documentation for Docker Swarm-etcd-Photon environment

The README.md file for the Docker Swarm-etcd-Photon OS environment is out of date and incorrect, and needs to be updated to reflect the current state of the learning environment.

Add a README to the Photon-Ansible learning environment

The VMware Photon-Ansible learning environment (found in photon-ansible) does not have a README file.

Linking to other peoples stuff?

Hey Scott,

This seems like a really cool idea! 😄 What do you think about linking to other peoples "tutorials" for distributed systems tools?

e.g.

• https://github.com/kelseyhightower/kubernetes-fleet-tutorial
• https://github.com/bketelsen/coreos-kubernetes-digitalocean

Add learning environment for containerd/runc

Add a learning environment that will enable users to work with/learn about runc (the future Docker container runtime) and containerd (the daemon that will manage runc).

Clean up Terraform+AWS environments

The Terraform+AWS environments need to be cleaned up a bit:

• Better consistency in filenames (use variables.tf everywhere)
• Consistent use of *.tfvars files to isolate variables and make environments more portable
• Expanded use of outputs to show users necessary information

The simple-ec2 environment has already been cleaned up, but the bastion-aws and new-vpc environments need to be reviewed. simple-module should adhere to this standards already.

vagrant-docker

On mac osx.

Command: "docker" "ps" "-a" "-q" "--no-trunc"

Stderr: Get http:///var/run/docker.sock/v1.19/containers/json?all=1: dial unix /var/run/docker.sock: permission denied. Are you trying to connect to a TLS-enabled daemon without TLS?

Any ideas?

dnf 2016 with virtualbox provisioner

Hi
I am trying to modify the Vagrant file of dnf-2016 for VirtualBox provider but I stumble across the
metadata of https://app.vagrantup.com/slowe/boxes/ubuntu-trusty-x64 which applies only for the vmware case. How can I overcome this ?

Add SSH key to Flatcar Linux on AWS Pulumi program

The Pulumi program found at flatcar/flatcar-pulumi-aws currently has a required configuration value that specifies the name of the AWS keypair. By modifying the program to create an SSH key and an AWS keypair, that required configuration value can be eliminated.

The ubuntu/ubuntu-pulumi-aws program already includes these resources and can be used as a model for the changes needed here.

Add CPU architecture support for Debian on Azure

The Pulumi program for running Debian on AWS (found in the debian/debian-pulumi-aws folder) has support for both AMD64- as well as ARM64-based instances. The Azure version (found in debian/debian-pulumi-azure) lacks this functionality.

Add an example of managing GitHub issue labels using Pulumi

Pulumi offers a GitHub provider that can be used for managing issue labels across repositories. I'd like to add an example of such a Pulumi program, written in Go, for others to use and/or learn from.

Add example of using Vagrant triggers and Docker contexts

I'd like to add an example of using Vagrant triggers to create/destroy Docker contexts automatically when a Vagrant machine is created or destroyed.

Running a Consul Cluster in Vagrant

I don’t see boostrap.json file. Will you be able to commit that file?

Add learning environment for OVS-Cumulus Linux integration

Add a learning environment for testing VTEP (VXLAN Tunnel End Point) connectivity between Cumulus Linux (via Cumulus VX) and Linux instances running Open vSwitch (OVS).

Add contribution guidelines

In order to make it easier for others to contribute to this repository, the repository needs some contribution guidelines.

Docker Macvlan not working using v.1.12.1

I forked a repo (2016-dnf-materials) to create a testbed scenario to play with docker networking modes: https://github.com/chadell/2016-demo-net-linux-networking
I tried to reproduce the same macvlan scenario you had there (and here) using Docker v.1.12.1 instead of the plugin I get that the container can't ping a remote host through the eth1 which is connected to the internal bridge created by Vagrant for all the VM hosts.
Debugging it using tcpdump on the container, docker-01 (host) and remote-01 (a remote host), I realize that I get ARP request and replies either in the remote-01 and the container, but when comes to ICMP I see the remote-01 receiving ICMP request from the container and replying but not ICMP requests coming into eth1 in the docker-01 host (before sending to the container), and neither works the ICMP request from the remote-01.

Config used in docker-01 host:
sudo ip link set eth1 promisc on
sudo docker network create -d macvlan --subnet=192.168.100.0/24 -o parent=eth1 pub_net
sudo docker run -td alpine /bin/sh
sudo docker network connect pub_net 2bab90874f2f

Tcpdump in remote-01 (192.168.100.110), the container has 192.168.100.2:

09:30:45.279510 ARP, Request who-has 192.168.100.2 tell 192.168.100.110, length 28
09:30:45.279812 ARP, Reply 192.168.100.2 is-at 02:42:c0:a8:64:02 (oui Unknown), length 46
09:30:45.279819 IP 192.168.100.110 > 192.168.100.2: ICMP echo request, id 1849, seq 1, length 64

Question about swarm & consul integration

Hi Scott

I have a little question about your blog

http://blog.scottlowe.org/2015/03/06/running-own-docker-swarm-cluster/

I understand the swarm setup, and it's supposed to listen on 2375 port instead of unix socket

But talking about the registrator, I find the --volumn option mapping the unix socket inside container, so when it comes with swarm nodes, how to map the :2375 to /tmp/docker.sock

Seems not working if you just use --volumn /var/run/docker.sock:/tmp/docker.sock

Add aws.instance_type=instances['instance_type'] to vagrant-aws/Vagrantfile

Add aws.instance_type=instances['instance_type'] to vagrant-aws/Vagrantfile because if not it will always use default values... (m3.medium) which is not the cheapest node to test ;)

Thnks,
Javier R.