traefik/mesh

Traefik Mesh - Simpler Service Mesh

Traefik (pronounced traffic) is a modern HTTP reverse proxy and load balancer that makes deploying microservices easy. Traefik integrates with your existing infrastructure components (Docker, Swarm mode, Kubernetes, Consul, Etcd, Rancher v2, Amazon ECS, ...) and configures itself automatically and dynamically. Pointing Traefik at your orchestrator should be the only configuration step you need.

---

. Overview . Features . Supported backends . Quickstart . Web UI . Documentation .

. Support . Release cycle . Contributing . Maintainers . Credits .

---

⚠️ Please be aware that the old configurations for Traefik v1.x are NOT compatible with the v2.x config as of now. If you're running v2, please ensure you are using a v2 configuration.

Overview

Imagine that you have deployed a bunch of microservices with the help of an orchestrator (like Swarm or Kubernetes) or a service registry (like etcd or consul). Now you want users to access these microservices, and you need a reverse proxy.

Traditional reverse-proxies require that you configure each route that will connect paths and subdomains to each microservice. In an environment where you add, remove, kill, upgrade, or scale your services many times a day, the task of keeping the routes up to date becomes tedious.

This is when Traefik can help you!

Traefik listens to your service registry/orchestrator API and instantly generates the routes so your microservices are connected to the outside world -- without further intervention from your part.

Run Traefik and let it do the work for you! (But if you'd rather configure some of your routes manually, Traefik supports that too!)

Features

• Continuously updates its configuration (No restarts!)
• Supports multiple load balancing algorithms
• Provides HTTPS to your microservices by leveraging Let's Encrypt (wildcard certificates support)
• Circuit breakers, retry
• See the magic through its clean web UI
• Websocket, HTTP/2, gRPC ready
• Provides metrics (Rest, Prometheus, Datadog, Statsd, InfluxDB 2.X)
• Keeps access logs (JSON, CLF)
• Fast
• Exposes a Rest API
• Packaged as a single binary file (made with ❤️ with go) and available as an official docker image

Supported Backends

• Docker / Swarm mode
• Kubernetes
• ECS
• File

Quickstart

To get your hands on Traefik, you can use the 5-Minute Quickstart in our documentation (you will need Docker).

Web UI

You can access the simple HTML frontend of Traefik.

Documentation

You can find the complete documentation of Traefik v2 at https://doc.traefik.io/traefik/.

A collection of contributions around Traefik can be found at https://awesome.traefik.io.

Support

To get community support, you can:

• join the Traefik community forum:

If you need commercial support, please contact Traefik.io by mail: mailto:[email protected].

Download

• Grab the latest binary from the releases page and run it with the sample configuration file:

./traefik --configFile=traefik.toml

• Or use the official tiny Docker image and run it with the sample configuration file:

docker run -d -p 8080:8080 -p 80:80 -v $PWD/traefik.toml:/etc/traefik/traefik.toml traefik

• Or get the sources:

git clone https://github.com/traefik/traefik

Introductory Videos

You can find high level and deep dive videos on videos.traefik.io.

Maintainers

We are strongly promoting a philosophy of openness and sharing, and firmly standing against the elitist closed approach. Being part of the core team should be accessible to anyone who is motivated and want to be part of that journey! This document describes how to be part of the maintainers' team as well as various responsibilities and guidelines for Traefik maintainers. You can also find more information on our process to review pull requests and manage issues in this document.

Contributing

If you'd like to contribute to the project, refer to the contributing documentation.

Please note that this project is released with a Contributor Code of Conduct. By participating in this project, you agree to abide by its terms.

Release Cycle

• We usually release 3/4 new versions (e.g. 1.1.0, 1.2.0, 1.3.0) per year.
• Release Candidates are available before the release (e.g. 1.1.0-rc1, 1.1.0-rc2, 1.1.0-rc3, 1.1.0-rc4, before 1.1.0).
• Bug-fixes (e.g. 1.1.1, 1.1.2, 1.2.1, 1.2.3) are released as needed (no additional features are delivered in those versions, bug-fixes only).

Each version is supported until the next one is released (e.g. 1.1.x will be supported until 1.2.0 is out).

We use Semantic Versioning.

Mailing Lists

• General announcements, new releases: mail at [email protected] or on the online viewer.
• Security announcements: mail at [email protected] or on the online viewer.

Credits

Kudos to Peka for his awesome work on the gopher's logo!.

The gopher's logo of Traefik is licensed under the Creative Commons 3.0 Attributions license.

The gopher's logo of Traefik was inspired by the gopher stickers made by Takuya Ueda. The original Go gopher was designed by Renee French.

mesh's Issues

Dashboard

Did we want a dashboard for this project? Or did we want to do something like have a deployable prometheus + grafana container sort of thing (like we do with our advocacy demos?)

Create Service Controller for Access Control

We need a service controller to manage Shadow services for access control.

This will allow SMI implementation of Access Control, and will may allow more features to be implemented in TrafficSplit etc.

Define MVP features for v1 launch

We should define a set of features that we want for v1 launch, so that we can ensure we have the features ready, and tests written to ensure correct behavior for these features.

Standardize project layout

Based on https://github.com/golang-standards/project-layout,

we have a non-standard layout.

We should improve by moving code into the standardized directories now

Handle services with multiple ports

We need to be able to create ingressRoutes(tcp) with different ports for each service port

Implement Prometheus, Grafana and Jaeger into i3o helm chart

We need to have the helm chart install:

• Prometheus
• Grafana
• Jaeger

And allow them to be configured from a default configuration to external sources if desired

Documentation

Eventually we are going to need documentation,

Because people.

Create Readme for installation

We need to have a Readme so that it is obvious how to install and configure i3o in a fresh cluster

Create Dockerfile and build for i3o

This needs to run in a docker container.

We need a dockerfile to make this work.

Core Functionality

There are a few things that this controller needs:

• Dockerization - #4
• CI integration - #11
• Tests - #29
• Helm chart - #16

Functionality requirements missing:

• Has to be able to CRUD Traefik CE KubernetesCRDs - #30
• Has to be able to interact with SMI CRDs - #66

Nice to haves:

• Be able to verify its own RBAC permissions to check if it has sufficient permissions - Not needed, since we do not want to grant access to rbac. Delegating to helm chart to ensure correct permissions
• Be able to run multiple replicas, and handle these CRUD operations atomically.

IngressRouteTCP has no middlewares

This means that IP filtering and whitelisting are not options that we have available to us.

I am going to speak with @juliens and @ldez about extending this with our provider in our i3o traefik implementation.

Create helm chart for i3o

We want to use helm to deploy i3o

We need a base helm chart to configure the install

Remove Traefik-mesh deployment code from controller

Since we are going to use helm, remove this code from controller

Implement the traefik CRD registration in helm chart

Move traefik mesh pods from http -> tcp mode

We don't want to use http for these pods.

We need to use TCP

Misnamed Imports

Since we have been using sources from different codebases, we need to standardize our imports so that we don't duplicate names.

	corev1 "k8s.io/api/core/v1"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	appsv1 "k8s.io/api/apps/v1"

Controller improvement/consolidation

Currently we use separate controllers for watching each type.

Once we get further along, we may not need separate controllers for each type, or we may be able to consolidate these controllers in a more efficient manner.

Convert from file configuration -> KubernetesCRD for config source

Instead of using a templated config file, we should move to CRD, which don't require templating files

Separate clusterRoleBinding into rolebinding

Many of the operations in the controller RBAC are due to updates in the mesh namespace

It might be nice to separate out the mesh namespace RBAC into a roleBinding,

and keep a much more minimal ClusterRoleBinding, since we don't need global permissions for everything in the cluster

Create integration test to handle a request

We should have an integration test that can handle an end-to-end request

SMI Traffic-Split Implementation

Traffic-split should be fairly easy to accomplish, once we have some weighting etc from Traefik v2.

Until then, we could do a janky workaround where we add the same route x/y times to mathematically make it work, but that is terrible

Discuss testing for i3o

How do we want to do tests?

How to define static config for I3o mesh nodes

Some of the traefik features (such as metrics) are static.

Did we want to manage these by a configmap (toml file)? or did we want to use CLI for everything?

CLI is so much cleaner than having to hunt down configmaps. We also don't have to worry about changes not propagating if we use CLI.

Demo Data

Do we need to have demo data in the app anymore?

Using "=" in flags breaks kubeconfig

Using:

./i3o patch --kubeconfig ~/.kube/config --master 127.0.0.1:9000 --debug
INFO[0000] Building Kubernetes Client...                
INFO[0000] Building Kubernetes CRD Client...            
INFO[0000] Preparing Cluster...         

Works, but:

./i3o patch --kubeconfig=~/.kube/config --master=127.0.0.1:9000 --debug
FATA[0000] Error building clients: stat ~/.kube/config: no such file or directory 

Breaks

Move Patch-DNS to a new command

We need to move the Patch DNS code to a new command, so that we can use separate RBAC permissions + jobify it.

Improve controller queues

Currently each controller has its own queue for processing events. We may want to have a global controller queue to be able to process so that we can move processing to a higher level controller (mesh controller)

Core pods all deleted at once

A regression was introduced in #34 that deletes all the core pods in the cluster.

This leads to total DNS outage for the entire cluster (including API).

I think it would be better to add a meta label to force a restart natively instead of deleting the pods.

Add Patch as a job with a separate serviceaccount in helm

We need:

1. A job
2. A new service account
3. New permissions via RBAC for this job

Tracing Implementation

How did we want to implement tracing?

Not sure how tracing is implemented in Traefik v2

Update meshservice/IngressRoute[TCP]s on user service update

We have create + delete, but we should handle updating.

Should also be able to have an integration test for this.

Review CRD for SMI

SMI is implemented via CRD, we need to get a controller for that

SMI Traffic-Access-Control Implementation

So it appears that traffic control via SMI is done by checking Service Accounts of running pods and tying them to destinations with running service accounts.

This is not difficult to do, but there are a bunch of ways we can accomplish it with traefik.

We can either do an IP whitelist, or we can do some other sort of filtering.

I have a few concerns:

1. If we are running in a large network, we may run into issues with large whitelists/blacklists
2. If we have a large dynamic network, there may be lots of changes that we have to watch for, and that may be a lot of load.

These concerns may not be big in reality, but just something to think about

Update CoreDNS patching mechanism

Right now we do a string replacement to patch in our coreDNS config.

We should look at having our own server block that we can just append to the data.

This would allow us to be compatible on more systems without risk of breaking things.

And be less janky.

Define label structure for configuration on services

We should define service labels to assist with defining routing.

This would allow us to start work on implementing features

Add required Match and Kind in ingressRoute creation

Match and Kind are not optional. They need to be defined.

Code de-dupe

There seems to be a bunch of duplicated code in the meshcontroller handler.

Look into using interfaces to allow for dedupe.

Service load balancing

We need to have load balancing in Traefik v2 before we can leverage it in i3o

Features and Integration tests

The following features and integration tests are needed:

• TCP Routing
  • Feature
  • Integration test
• HTTP Routing
  • Feature
  • Integration test
• Metrics - #65
  • Feature
  • Integration test
• Tracing - #63
  • Feature
  • Integration test
• Service load balancing - #64
  • Feature
  • Integration test
• Routing Rules - This one could be combined with TCP and HTTP, since they will both use rules, however this is also possibly done part of the traffic routing part of SMI
  • Feature
  • Integration test
• Retries - #166
  • Feature
  • Integration test
• Failover
  • Feature
  • Integration test
• Access control
  • Feature
  • Integration test
• Rate limit - #168
  • Feature
  • Integration test
• circuit breakers - #167
  • Feature
  • Integration test

Follow up RBAC Updates for patch/controller

We need to verify that the RBAC matches the client access requirements

Verify Patch service account is deleted when controller starts

We use a high-perms RBAC SA for patching DNS.

We should confirm that it is deleted as part of the controller verification on startup

Log warning when UDP port is detected

Since we don't have UDP support at this time, we should be logging a warning, and skipping the port

Add ability to create KubernetesCRD from Traefik CE

We need to be able to CRUD the Traefik CE KubernetesCRDs

Investigate Reverse DNS lookup on IP to get pod for SMI

We could look into using reverse IP lookups on the source IP to get the pod reference.

this may allow us to not have to deal with IP whitelisting for source filtering

Route by service name and Service IP

We should route by not only service name (DNS) but also service IP.

This is for both TCP and HTTP

Use TCP routing by default

Since services are TCP by default, we should also use TCP routing by default

Ignore `default/kubernetes` service

The kubernetes API service should not be managed by mesh.

We should add it to the list of namespaces to ignore.

Rename Verify methods to CRUD equivalents

Although verifiyExists is nice,

create is clearer. With a comment that if exists, returns the object.

Related to #57

Create label for ingressRoute selection to avoid external routing objects

We should use:

i3o-mesh:internal as the label

Metrics implementation

Once metrics are enabled in traefik v2, we need to have them implemented in i3o. This will also be required for SMI