Giter Site home page Giter Site logo

alexellis / faas-containerd Goto Github PK

View Code? Open in Web Editor NEW
61.0 8.0 10.0 5.2 MB

containerd and CNI provider for OpenFaaS

Home Page: https://blog.alexellis.io/faas-containerd-serverless-without-kubernetes/

License: MIT License

Go 97.87% Makefile 2.13%
containerd serverless faasd openfaas

faas-containerd's Introduction

faas-containerd

Build Status

OpenFaaS provider for containerd - single node / edge workloads

What's the use-case?

OpenFaaS providers can be built for any backend, even for an in-memory datastore. Some users could benefit from a lightweight, single-node execution environment. Using containerd and bypassing Kubernetes or Docker should reduce the start-time for functions and allow for running in resource-constrained environments.

Pros:

  • Fast cold-start
  • containerd features available such as pause/snapshot
  • Super lightweight
  • Basic service-discovery and inter-service communication through /etc/hosts and bridge

Cons:

  • No clustering (yet)

Status

Proof of concept.

This project implements the faas-provider SDK.

Architecture

faas-provider conceptual architecture

See other examples:

Goals:

  • Deploy container specified via PUT to /system/functions
  • Serve HTTP traffic from deployed container via /function/NAME
  • List running containers via GET on /system/functions
  • Clean-up containers on exit
  • Give configuration for running faas-containerd / OpenFaaS gateway and Prometheus via systemd unit files or similar

An update - Jan 2020

faas-containerd is now part of the faasd project, which adds additional orchestration to bring the whole OpenFaaS experience and ecosystem to containerd.

See my tutorial on how to get started with faasd. It is written for RPi users, but faasd is designed to be portable and also runs on x86_64 and ARM64:

Demo

Test it out

You need a Linux computer, VM, or bare-metal cloud host.

Get some build dependencies

I used Ubuntu 18.04 LTS on Packet.com using the c1.small.x86 host. You can use multipass.run to get an Ubuntu host on any OS - Windows, MacOS, or Linux.

sudo apt update && \
  sudo apt install -qy runc \
  	bridge-utils \
	tmux git \
  	build-essential \
	libbtrfs-dev libseccomp-dev

Install Go 1.12 (x86_64)

curl -SLsf https://dl.google.com/go/go1.12.14.linux-amd64.tar.gz > go.tgz
sudo rm -rf /usr/local/go/
sudo mkdir -p /usr/local/go/
sudo tar -xvf go.tgz -C /usr/local/go/ --strip-components=1

export GOPATH=$HOME/go/
export PATH=$PATH:/usr/local/go/bin/

go version

Or on Raspberry Pi (armhf)

curl -SLsf https://dl.google.com/go/go1.12.14.linux-armv6l.tar.gz > go.tgz
sudo rm -rf /usr/local/go/
sudo mkdir -p /usr/local/go/
sudo tar -xvf go.tgz -C /usr/local/go/ --strip-components=1

export GOPATH=$HOME/go/
export PATH=$PATH:/usr/local/go/bin/

go version

Get containerd

  • Install containerd (or build from source)

Note: This can only be run on x86_64

export VER=1.3.2
curl -sLSf https://github.com/containerd/containerd/releases/download/v$VER/containerd-$VER.linux-amd64.tar.gz > /tmp/containerd.tar.gz \
  && sudo tar -xvf /tmp/containerd.tar.gz -C /usr/local/bin/ --strip-components=1

containerd -version
  • Or clone / build / install containerd from source:
export GOPATH=$HOME/go/
mkdir -p $GOPATH/src/github.com/containerd
cd $GOPATH/src/github.com/containerd
git clone https://github.com/containerd/containerd
cd containerd
git fetch origin --tags
git checkout v1.3.2

make
sudo make install

containerd --version

Kill any old containerd version:

# Kill any old version
sudo killall containerd
sudo systemctl disable containerd

Start containerd in a new terminal:

sudo containerd &

CNI Plugins

Install the CNI plugins like this:

export CNI_VERSION=v0.8.2
  • For PC run export ARCH=amd64
  • For RPi/armhf run export ARCH=arm
  • For arm64 run export ARCH=arm64

Then run:

mkdir -p /opt/cni/bin
curl -sSL https://github.com/containernetworking/plugins/releases/download/${CNI_VERSION}/cni-plugins-linux-${ARCH}-${CNI_VERSION}.tgz | tar -xz -C /opt/cni/bin

Check it worked:

/opt/cni/bin/bridge --help

CNI bridge plugin v0.8.2

Enable forwarding

This is required to allow containers in containerd to access the Internet via your computer's primary network interface.

sudo /sbin/sysctl -w net.ipv4.conf.all.forwarding=1

Make the setting permanent:

echo "net.ipv4.conf.all.forwarding=1" | sudo tee -a /etc/sysctl.conf

Build and run faas-containerd

  • Get a binary

     # For x86_64
     sudo curl -fSLs "https://github.com/alexellis/faas-containerd/releases/download/0.4.0/faas-containerd" \
       -o "/usr/local/bin/faas-containerd" \
       && sudo chmod a+x "/usr/local/bin/faas-containerd"
    
     # armhf
     sudo curl -fSLs "https://github.com/alexellis/faas-containerd/releases/download/0.4.0/faas-containerd-armhf" \
       -o "/usr/local/bin/faas-containerd" \
       && sudo chmod a+x "/usr/local/bin/faas-containerd"
    
     # arm64
     sudo curl -fSLs "https://github.com/alexellis/faas-containerd/releases/download/0.4.0/faas-containerd-arm64" \
       -o "/usr/local/bin/faas-containerd" \
       && sudo chmod a+x "/usr/local/bin/faas-containerd"
    
     # run faas-containerd
     sudo service_timeout=1m ./faas-containerd
  • Or build from source

     export GOPATH=$HOME/go/
    
     mkdir -p $GOPATH/src/github.com/alexellis/faas-containerd
     cd $GOPATH/src/github.com/alexellis/faas-containerd
     git clone https://github.com/alexellis/faas-containerd
     cd faas-containerd
    
     go build && sudo service_timeout=1m ./faas-containerd

Listens on port TCP/8081

Test out your faas-containerd

Get the OpenFaaS CLI:

curl -sLfS https://cli.openfaas.com | sudo sh

Deploy a function with a server

faas store deploy figlet -g 127.0.0.1:8081 --update=true --replace=false

Deploy a ping function with a server

faas-cli deploy --image alexellis2/ping:0.1 \
  -g 127.0.0.1:8081 --update=true --replace=false --name ping

Deploy nodeinfo function with a server

faas-cli store deploy nodeinfo \
  -g 127.0.0.1:8081 --update=true --replace=false

Try to list functions:

faas-cli list -g 127.0.0.1:8081

Get a function's status:

faas-cli describe nodeinfo -g 127.0.0.1:8081

Try to invoke a function:

echo "-c 1 8.8.8.8" | faas-cli invoke ping -g 127.0.0.1:8081

echo "verbose" | faas-cli invoke nodeinfo -g 127.0.0.1:8081

List containers with ctr:

sudo ctr --namespace openfaas-fn container list

Delete container, snapshot and task:

sudo ctr --namespace openfaas-fn task kill figlet
sudo ctr --namespace openfaas-fn task delete figlet
sudo ctr --namespace openfaas-fn container delete figlet
sudo ctr --namespace openfaas-fn snapshot remove figlet-snapshot

Enable Basic Auth

Basic auth requires three things: a username file, a password file, and an env variable.

First, create the username and password, make sure to pick a secure password for production deployments:

echo admin >> basic-auth-user
echo localdev >> basic-auth-password

Then set the basic_auth and secret_mount_path env variables when starting faas-containerd:

service_timeout=1m basic_auth=true secret_mount_path=`pwd` sudo ./bin/faas-containerd

The default value for secret_mount_path is /run/secrets/.

Once it is running, you can quickly verify that auth is being enforced by using the CLI:

$ faas-cli list -g 127.0.0.1:8081
Unauthorized access, run "faas-cli login" to setup authentication for this server

Now login:

cat basic-auth-password | faas-cli login -g 127.0.0.1:8081 --tls-no-verify -u admin --password-stdin

You can now deploy and invoke a function as before:

faas-cli deploy --image alexellis2/ping:0.1 \
  -g 127.0.0.1:8081 --update=true --replace=false --name ping
echo "-c 1 8.8.8.8" | faas-cli invoke ping -g 127.0.0.1:8081

The auth is saved in ~/.openfaas/config.yml.

Links

License

MIT

faas-containerd's People

Contributors

alexellis avatar carlosedp avatar darkflib avatar lucasroesler avatar utsavanand2 avatar waterdrips avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

faas-containerd's Issues

Add labels to the containers from the function spec

Add labels to the containers from the function spec

Expected Behaviour

If you create a function with a label, that label should be added to the container definition, then read back later in the list endpoint.

Current Behaviour

They are ignored

Possible Solution

I'm OK if we only store the label and don't read it back for the first PR.

Update https://github.com/alexellis/faas-containerd/blob/master/handlers/update.go and https://github.com/alexellis/faas-containerd/blob/master/handlers/deploy.go

The types.FunctionDeployment struct already contains labels.

Steps to Reproduce (for bugs)

  1. faas-cli store deploy figlet --label has=label
  2. faas-cli describe figlet

The above on faas-netes / K8s will show the labels, on faas-containerd it won't

Unknown error running netns on riscv64 architecture

I've successfully built faas-containerd on Risc-V architecture (submitted PR #1) but got the following error:

On console 1:

❯ sudo function_uptime=120m ./faas-containerd
2019/12/20 00:00:02 faas-containerd starting..
2019/12/20 00:00:02 TCP port: 8081	Timeout: 1m0s	Function uptime: 2h0m0s
{"service":"figlet","image":"carlosedp/faas-figlet:riscv64","network":"","envProcess":"","envVars":{},"constraints":[],"secrets":[],"labels":{},"annotations":{},"limits":null,"requests":null,"readOnlyRootFilesystem":false}


# This happens after the faas-cli deploy command from console 2

2019/12/20 00:00:08 docker.io/carlosedp/faas-figlet:riscv64
2019/12/20 00:00:08 113717265 <nil>
2019/12/20 00:00:10 OCI runtime create failed: error executing hook /usr/bin/netns (exit code: 1): unknown

Console 2:

❯ faas-cli deploy --image=carlosedp/faas-figlet:riscv64 --name=figlet -g 127.0.0.1:8081 --update=true --replace=false
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.
Function figlet already exists, attempting rolling-update.

Deployed. 200 OK.
URL: http://127.0.0.1:8081/function/figlet

Netns was also bumped and built successfully. I've submitted the PR genuinetools/netns#21 addressing it's requirement bumps.

My stack is containerd -> crun (pure C runtime). Runs fine on Docker and Podman:

❯ containerd --version
containerd ./cmd/containerd v1.3.1 c7a4f874b3267c499484aae602d1257b12d69e40

❯ crun --version
crun version 0.10.3.56-ffaa
commit: ffaa55bb420bce27496fdc1201e5b219f31c5ead
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

Any tips on how to debug it?
Cc. @jessfraz

faas-containerd panics with a runtime error: index out of range

Expected Behaviour

I shoud be able to deploy my function with faas-cli

Current Behaviour

faas-containerd is crashing with an index out of range error

Screenshot 2020-01-11 at 1 29 11 AM

Steps to Reproduce (for bugs)

Followed the steps in the README.md

Your Environment

  • OS and architecture:
    Ubuntu 18.04

  • Versions:

go version
go not installed

sudo /opt/cni/bin/bridge version
CNI bridge plugin v0.8.4

containerd -version
containerd github.com/containerd/containerd v1.3.2 ff48f57fc83a8c44cf4ad5d672424a98ba37ded6

uname -a
Linux faas 5.0.0-1028-gcp #29~18.04.1-Ubuntu SMP Thu Dec 19 19:04:05 UTC 2019 x86_64 
x86_64 x86_64 GNU/Linux

cat /etc/os-release

Screenshot 2020-01-11 at 1 34 15 AM

Report containers as found in containerd rather than in-memory

Report containers as found in containerd rather than in-memory

Expected Behaviour

Any containers with matching tasks and IPs should be reported on the list functions handler.

Current Behaviour

We only report from an in-memory map rather than going to containerd.

Possible Solution

The pattern would be in pseudocode, something like this:

def List_containers_and_IPs() :
res = []
cs=  containerd.listcontainers(namespace)
for c in cs:
   t = containerd.get_task(c)
   pid = t.pid
  ip = CNI or Weavenet . GetIP(pid)
  res = append({name: cs.name, ip: ip })
return res

This change would mean that containers could survive a restart of faas-containerd, but right now, since we only track in memory, we lose that data when the process starts up again.

Caching may be required if the list function is slow.

If the machine reboots, only the container survives, but not the task and the task is the process with a PID and IP. Therefore if we have only a container and no task, we could report 0/0 replicas and if invoked, scale to 1 should create any missing task.

Scaling to zero is supported in this provider, but the function status endpoint is hard-coded to 1 replica, the above work will fix that and allow 0/0 to be reported when that's the case.

Update README for latest version

There are two minor issues in the readme...

The first is a missing faas-containerd start in the primary flow.
The second is an omitted container when using ctr to list containers.

Expected Behaviour

Current Behaviour

Possible Solution

PR 14 fixes these

Steps to Reproduce (for bugs)

Context

Your Environment

  • OS and architecture:

  • Versions:

go version

containerd -version

uname -a

cat /etc/os-release

Removing a function with active replicas (task running) fails

Trying to remove a function that is currently running (has a RUNNING task), fails.

After trying to remove again, it succeeds.

Expected Behaviour

Function gets removed with first command.

Current Behaviour

Running function:

❯ faas store deploy figlet --name=figlet2
WARNING! Communication is not secure, please consider using HTTPS. Letsencrypt.org offers free SSL/TLS certificates.

Deployed. 200 OK.
URL: http://localhost:8081/function/figlet2

❯ sudo ctr -n openfaas-fn container ls
CONTAINER    IMAGE                                RUNTIME
figlet2      docker.io/functions/figlet:0.13.0    io.containerd.runc.v2


❯ sudo ctr -n openfaas-fn task ls
TASK       PID     STATUS
figlet2    7252    RUNNING

Deploy logs:

Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 16:42:43 [Update] request: {"service":"figlet2","image":"functions/figlet:0.13.0","network":"","envProcess":"figlet","envVars":{},"constraints":[],"secrets":[],"labels":{},"annotations":{},"limits":null,"requests":null,"readOnlyRootFilesystem":false}
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 16:42:43 [Update] service figlet2 not found
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 16:42:43 [Deploy] request: {"service":"figlet2","image":"functions/figlet:0.13.0","network":"","envProcess":"figlet","envVars":{},"constraints":[],"secrets":[],"labels":{},"annotations":{},"limits":null,"requests":null,"readOnlyRootFilesystem":false}
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 16:42:43 Deploy docker.io/functions/figlet:0.13.0 size: 5658006
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 16:42:43 Container ID: figlet2        Task ID figlet2:        Task PID: 7252
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 16:42:43 figlet2 has IP: 10.62.0.163.
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 21:42:43 Version: 0.13.0        SHA: fa93655d90d1518b04e7cfca7d7548d7d133a34e
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 21:42:43 Read/write timeout: 5s, 5s. Port: 8080
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 21:42:43 Writing lock-file to: /tmp/.lock
Jan 17 16:42:43 debian10 faas-containerd[6669]: 2020/01/17 21:42:43 Metrics server. Port: 8081

Trying to remove:

❯ faas-cli remove figlet2
Deleting: figlet2.
Server returned unexpected status code 500 error deleting container figlet2, figlet2, cannot delete running task figlet2: failed precondition

Logs:

Jan 17 16:44:16 debian10 faas-containerd[6669]: 2020/01/17 16:44:16 [Delete] request: {"functionName":"figlet2"}
Jan 17 16:44:16 debian10 faas-containerd[6669]: 2020/01/17 16:44:16 [Delete] removing CNI network for figlet2
Jan 17 16:44:16 debian10 faas-containerd[6669]: 2020/01/17 16:44:16 [Delete] removed figlet2 with namespace /proc/7252/ns/net and ID figlet2-7252
Jan 17 16:44:16 debian10 faas-containerd[6669]: Status of figlet2 is: running
Jan 17 16:44:16 debian10 faas-containerd[6669]: 2020/01/17 16:44:16 Need to kill figlet2
Jan 17 16:44:16 debian10 faas-containerd[6669]: 2020/01/17 21:44:16 SIGTERM received.. shutting down server in 5s
Jan 17 16:44:16 debian10 faas-containerd[6669]: 2020/01/17 21:44:16 Removing lock-file : /tmp/.lock
Jan 17 16:44:21 debian10 faas-containerd[6669]: 2020/01/17 21:44:21 No new connections allowed. Exiting in: 5s
Jan 17 16:44:21 debian10 faas-containerd[6669]: 2020/01/17 16:44:21 [Delete] error removing figlet2, error deleting container figlet2, figlet2, cannot delete running task figlet2: failed precondition

Task gets stopped but container is not removed:

❯ sudo ctr -n openfaas-fn container ls
CONTAINER    IMAGE                                RUNTIME
figlet2      docker.io/functions/figlet:0.13.0    io.containerd.runc.v2

❯ sudo ctr -n openfaas-fn task ls
TASK       PID     STATUS

figlet2    7252    STOPPED

Running remove command again, removes:

❯ faas-cli remove figlet2
Deleting: figlet2.
Removing old function.

Logs:

Jan 17 16:45:29 debian10 faas-containerd[6669]: 2020/01/17 16:45:29 [Delete] request: {"functionName":"figlet2"}
Jan 17 16:45:29 debian10 faas-containerd[6669]: Status of figlet2 is: stopped
Jan 17 16:45:29 debian10 faas-containerd[6669]: 2020/01/17 16:45:29 Need to kill figlet2
Jan 17 16:45:29 debian10 faas-containerd[6669]: 2020/01/17 16:45:29 [Delete] deleted figlet2

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

  • OS and architecture:

  • Versions:

go version

containerd -version

uname -a

cat /etc/os-release

Functions fail to deploy

Expected Behaviour

The functions should get deployed on a faas deploy invocation

Current Behaviour

I'm getting 400 status response code from faas-containerd
Screenshot 2020-01-09 at 12 14 39 PM

Possible Solution

It looks like it is still looking for netns not CNI

Steps to Reproduce (for bugs)

I followed the steps in the README.

Your Environment

  • OS and architecture: Debian 9 stretch

  • Versions:

go version: 
go version go1.12.14 linux/amd64

containerd -version
containerd github.com/containerd/containerd v1.3.2 ff48f57fc83a8c44cf4ad5d672424a98ba37ded6

uname -a
Linux faas-containerd 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64 GNU/Linux

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Containers with stopped tasks cannot be scaled up

If a container has it's task as "STOPPED", scaling it back generates an error and the task does not run.

Expected Behaviour

Have the task in RUNNING state.

Current Behaviour

Show function and it's container and task:

❯ faas-cli list
Function                      	Invocations    	Replicas
figlet1                       	0              	0

❯ sudo ctr -n openfaas-fn container ls
CONTAINER    IMAGE                                RUNTIME
figlet1      docker.io/functions/figlet:0.13.0    io.containerd.runc.v2

❯ sudo ctr -n openfaas-fn task ls
TASK       PID     STATUS
figlet1    6760    STOPPED

Try to scale it back to 1 replica:

❯ curl http://127.0.0.1:8081/system/scale-function/figlet1 -d '{"serviceName":"figlet1", "replicas": 1}'
cannot scale task for figlet1, error: cannot resume a stopped container: unknown

Generates error:

Jan 17 16:32:51 debian10 faas-containerd[6669]: 2020/01/17 16:32:51 Resolve: "figlet1"
Jan 17 16:32:51 debian10 faas-containerd[6669]: 2020/01/17 16:32:51 error with proxy request to: http://:8080, Post http://:8080: dial tcp :8080: connect: connection refused
Jan 17 16:32:56 debian10 faas-containerd[6669]: 2020/01/17 16:32:56 [Scale] request: {"serviceName":"figlet1", "replicas": 1}
Jan 17 16:32:56 debian10 faas-containerd[6669]: 2020/01/17 16:32:56 [Scale] cannot scale task for figlet1, error: cannot resume a stopped container: unknown

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

  • OS and architecture:

Latest built from persistence branch.

  • Versions:
go version

containerd -version

uname -a

cat /etc/os-release

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.