edgenet-project/node

Ansible playbooks to deploy EdgeNet nodes.

EdgeNet Node Setup

📌 For instructions on how to use and how to contribute a node to EdgeNet, please see the EdgeNet website.

Architecture

This repository contains the code necessary to automatically deploy an EdgeNet node. It consists of three parts:

1. The bootstrap script which installs Ansible and runs the node playbook with ansible-pull.
2. The Ansible roles, which setups SSH access, containerd, Kubernetes and the EdgeNet service.
3. The EdgeNet service, which is run on every boot, via systemd, to configure the node hostname and network. It will also join the node to the cluster, if not already joined.

In most cases users will run the bootstrap script and wait until the node is ready. However, it is also possible to run the node playbook directly on the target machines with ansible-playbook.

Bootstrap script

The bootstrap.sh script installs Ansible and Git, and runs the node playbook. It can be configured with the following environment variables:

Name	Default	Description
EDGENET_ASK_CONFIRMATION	1	Whether to ask to continue or not.
EDGENET_PLAYBOOK	edgenet-node.yml	Name of the playbook to run.
EDGENET_REF	main	Git reference to use.
EDGENET_REPOSITORY	https://github.com/EdgeNet-project/node.git	URL of the Git repository containing the playbook to run.

Ansible roles

edgenet-ssh

Create an EdgeNet user with SSH access and passwordless sudo.

Variable	Default	Description
edgenet_ssh_user	edgenet	EdgeNet SSH user
edgenet_ssh_port_alt	25010	Alternative SSH port if port 22 is unavailable
edgenet_ssh_public_key	[...] edgenet.planet-lab.eu (2021)	Public SSH key of the EdgeNet user

edgenet-kubernetes

Setup Docker and Kubernetes.

Variable	Default	Description
edgenet_service_state	restarted	State of the EdgeNet systemd service
edgenet_node_version	-	The release of the EdgeNet service to install
containerd_version	-	The containerd version to install
kubernetes_version	-	The kubernetes version to install

EdgeNet service

The EdgeNet service is written in Go, in the main.go file and the pkg/ directory.

Development

Run the local bootstrap script with the local Ansible playbook

git clone [email protected]:EdgeNet-project/node.git && cd node
env EDGENET_REF="$(git rev-parse HEAD)" EDGENET_REPOSITORY="file://$(pwd)" ./bootstrap.sh

Run the Ansible playbook locally

git clone [email protected]:EdgeNet-project/node.git && cd node
ansible-playbook --connection=local --inventory localhost, edgenet-node.yml

Use the Ansible playbook from a specific branch

export EDGENET_REF=my-branch
bash -ci "$(wget -O - https://raw.githubusercontent.com/EdgeNet-project/node/${EDGENET_REF}/bootstrap.sh)"

Update the node binary

1. Create a GitHub release and wait for the completion of the associated workflow
2. Update edgenet_node_version in vars/edgenet-production.yml

Alternatively, for debugging, you can compile the node binary locally and directly upload it to /opt/edgenet/node on the remote node.

Cluster-wide play

To run a playbook across all the nodes of the cluster:

kubectl get nodes -l node-role.kubernetes.io/control-plane!= -o json | jq -r '.items[].status.addresses[0].address' > nodes.ini
ansible-playbook -i nodes.ini ...

Contributing

The EdgeNet software is free and open source, licensed under the Apache 2.0 license; we invite you to contribute. For more information, see EdgeNet-project/edgenet.

node's Issues

TASK [edgenet-kubernetes : Ensure Kubernetes APT GPG key is present] is failing on Ubuntu 22.04.2

When running the boostrap.sh script, the task installing the GPG key of the kubernetes repo is failing with:

TASK [edgenet-kubernetes : Ensure Kubernetes APT repository is present] ********
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: W:https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details., W:GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY B53DC80D13EDEF05, E:The repository 'http://apt.kubernetes.io kubernetes-xenial InRelease' is not signed."}

Allow users to specify their node location

Handle ARM64 platforms

Ubuntu 20.04
Everything seems to work on Ubuntu 20.04, excepted for the Docker installation.
Need to set arm64 in https://github.com/geerlingguy/ansible-role-docker.

CentOS
Need to set aarch64 for https://github.com/geerlingguy/ansible-role-kubernetes

Install on Fedora 38 fails because of containerd version

I have tried to follow the steps on https://www.edge-net.org/pages/node-contribution.html in order to contribute a node, and chose to use Fedora 38.
The installation fails because the containerd version offered on Fedora after the repo https://download.docker.com/linux/fedora/38/ is setup is version 1.6 whereas the var file references the version 1.5.

Is that expected ? Is there a known workaround ?

edgenet.kubernetes: Fix Docker installation

{
   "cache_update_time":1614770720,
   "cache_updated":false,
   "changed":false,
   "msg":"'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\"      install 'docker-ce=5:19.03*'' failed: E: Sub-process /usr/bin/dpkg returned an error code (1)\n",
   "rc":100,
   "stderr":"E: Sub-process /usr/bin/dpkg returned an error code (1)\n",
   "stderr_lines":[
      "E: Sub-process /usr/bin/dpkg returned an error code (1)"
   ],
   "stdout":"Reading package lists...\nBuilding dependency tree...\nReading state information...\ndocker-ce is already the newest version (5:19.03.15~3-0~ubuntu-focal).\nThe following packages were automatically installed and are no longer required:\n  libfprint-2-tod1 libllvm9 nvidia-kernel-common-440\nUse 'sudo apt autoremove' to remove them.\n0 upgraded, 0 newly installed, 0 to remove and 42 not upgraded.\n1 not fully installed or removed.\nAfter this operation, 0 B of additional disk space will be used.\nSetting up docker-ce (5:19.03.15~3-0~ubuntu-focal) ...\r\nJob for docker.service failed because the control process exited with error code.\r\nSee \"systemctl status docker.service\" and \"journalctl -xe\" for details.\r\ninvoke-rc.d: initscript docker, action \"start\" failed.\r\n* docker.service - Docker Application Container Engine\r\n     Loaded: loaded (\u001b]8;;file://for1452/lib/systemd/system/docker.service\u0007/lib/systemd/system/docker.service\u001b]8;;\u0007; enabled; vendor preset: enabled)\r\n     Active: activating (auto-restart) (Result: exit-code) since Wed 2021-03-03 12:35:33 CET; 5ms ago\r\nTriggeredBy: * docker.socket\r\n       Docs: \u001b]8;;https://docs.docker.com\u0007https://docs.docker.com\u001b]8;;\u0007\r\n    Process: 279658 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock \u001b[0;1;31m(code=exited, status=1/FAILURE)\u001b[0m\r\n   Main PID: 279658 (code=exited, status=1/FAILURE)\r\ndpkg: error processing package docker-ce (--configure):\r\n installed docker-ce package post-installation script subprocess returned error exit status 1\r\nErrors were encountered while processing:\r\n docker-ce\r\n",
   "stdout_lines":[
      "Reading package lists...",
      "Building dependency tree...",
      "Reading state information...",
      "docker-ce is already the newest version (5:19.03.15~3-0~ubuntu-focal).",
      "The following packages were automatically installed and are no longer required:",
      "  libfprint-2-tod1 libllvm9 nvidia-kernel-common-440",
      "Use 'sudo apt autoremove' to remove them.",
      "0 upgraded, 0 newly installed, 0 to remove and 42 not upgraded.",
      "1 not fully installed or removed.",
      "After this operation, 0 B of additional disk space will be used.",
      "Setting up docker-ce (5:19.03.15~3-0~ubuntu-focal) ...",
      "Job for docker.service failed because the control process exited with error code.",
      "See \"systemctl status docker.service\" and \"journalctl -xe\" for details.",
      "invoke-rc.d: initscript docker, action \"start\" failed.",
      "* docker.service - Docker Application Container Engine",
      "     Loaded: loaded (\u001b]8;;file://for1452/lib/systemd/system/docker.service\u0007/lib/systemd/system/docker.service\u001b]8;;\u0007; enabled; vendor preset: enabled)",
      "     Active: activating (auto-restart) (Result: exit-code) since Wed 2021-03-03 12:35:33 CET; 5ms ago",
      "TriggeredBy: * docker.socket",
      "       Docs: \u001b]8;;https://docs.docker.com\u0007https://docs.docker.com\u001b]8;;\u0007",
      "    Process: 279658 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock \u001b[0;1;31m(code=exited, status=1/FAILURE)\u001b[0m",
      "   Main PID: 279658 (code=exited, status=1/FAILURE)",
      "dpkg: error processing package docker-ce (--configure):",
      " installed docker-ce package post-installation script subprocess returned error exit status 1",
      "Errors were encountered while processing:",
      " docker-ce"
   ]
}

edgenet.ssh: fix install of public SSH key on Google Cloud

Use API/instance metadata

Update VPN peer endpoint when public IP changes

• How to update securely the VPN peer object?

The NodeContribution controller is stuck "In Progress"

edgenet.ssh: add second SSH port (25010)

edgenet.kubernetes: autodetect SSH port in join-cluster.sh

• Use edgenet_ssh_port_alt?
• Find Port ** in /etc/ssh/sshd_config?

edgenet.kubernetes: fix hostname provider detection

+ . /opt/edgenet/common.sh
+ [ ! -f /opt/edgenet/hostname ]
+ rand a-f0-9 4
+ dd bs=512 count=1 if=/dev/urandom
+ tr --complement --delete a-f0-9
+ fold --width 4
+ head --lines 1
1+0 records in
1+0 records out
512 bytes copied, 2,6051e-05 s, 19,7 MB/s
+ hash=14fd
+ aws
+ get http://169.254.169.254/latest/meta-data/
+ curl --ipv4 --fail --max-time 2 --noproxy --silent --show-error --header Metadata: true --header Metadata-Flavor: Google http://169.254.169.254/latest/meta-data/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (28) Connection timed out after 2000 milliseconds
+ gcp
+ get http://metadata.google.internal/computeMetadata/v1/instance/
+ curl --ipv4 --fail --max-time 2 --noproxy --silent --show-error --header Metadata: true --header Metadata-Flavor: Google http://metadata.google.internal/computeMetadata/v1/insta
nce/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: metadata.google.internal
+ scw
+ get http://169.254.42.42/conf
+ curl --ipv4 --fail --max-time 2 --noproxy --silent --show-error --header Metadata: true --header Metadata-Flavor: Google http://169.254.42.42/conf
+ grep
+ cut --delimiter = --fields 2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:02 --:--:--     0
curl: (28) Connection timed out after 2000 milliseconds
+ scw LOCATION_ZONE_ID
#!/bin/sh
+ get http://169.254.42.42/conf
+ curl --ipv4 --fail --max-time 2 --noproxy --silent --show-error --header Metadata: true --header Metadata-Flavor: Google http://169.254.42.42/conf
+ grep LOCATION_ZONE_ID
+ cut --delimiter = --fields 2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (7) Failed to connect to 169.254.42.42 port 80: No route to host
+ region=
+ hostname=scw--14fd
+ echo scw--14fd.edge-net.io
+ cat /opt/edgenet/hostname
+ hostname=scw--14fd.edge-net.io
+ hostnamectl set-hostname scw--14fd.edge-net.io
+ sed -i.bak /.edge-net.io$/d /etc/hosts
+ echo 127.0.0.1 scw--14fd.edge-net.io
+ tee -a /etc/hosts
127.0.0.1 scw--14fd.edge-net.io

Include MAC address in node name for Odroid boards

edgenet.ssh: Handle already existing edgenet user

Set --resolv-conf for nodes running systemd-resolved

https://coredns.io/plugins/loop/#troubleshooting-loops-in-kubernetes-clusters

/run/systemd/resolve/resolv.conf