Giter Site home page Giter Site logo

stellar's Introduction

   _____ __       ____
  / ___// /____  / / /___ ______
  \__ \/ __/ _ \/ / / __ `/ ___/
 ___/ / /_/  __/ / / /_/ / /
/____/\__/\___/_/_/\__,_/_/

Simplified Container Runtime Cluster

Stellar is designed to provide simple container runtime clustering. One or more nodes are joined together to create a cluster. The cluster is eventually consistent making it ideal for transient workloads or edge computing where nodes are not always guaranteed to have high bandwidth, low latency connectivity.

Why

There are several container platforms and container orchestrators out there. However, they are too complex for my use. I like simple infrastructure that is easy to deploy and manage, tolerates failure cases and is easy to debug when needed. With the increased tolerance in failure modes, this comes at a consistency cost. It may not be for you. Use the best tool for your use case. Enjoy :)

Features

  • Container execution via containerd
  • Multihost Networking via CNI
  • Service Discovery via DNS
  • Cluster event system via NATS
  • Builtin Proxy using Radiant (zero downtime reloads, canary deploys, health checks, automatic HTTPS)
  • Masterless design
  • Efficient use of system resources
  • Simple daemon deployment

Downloads

For official releases, see the Releases

You can also grab the latest Master Build

Building

In order to build Stellar you will need the following:

Once you have the requirements you can build.

If you change / update the protobuf definitions you will need to generate:

make generate

To build the binaries (client and server) run:

make

Docker

Alternatively you can use Docker to build:

To generate protobuf:

make docker-generate

To build binaries:

make docker-build

Running

To run Stellar, once you have a working containerd installation follow these steps:

  • Install Containerd version >=1.1
  • Build binaries or get a release
  • Copy /bin/sctl to /usr/local/bin/
  • Copy /bin/stellar to /usr/local/bin/
  • Copy /bin/stellar-cni-ipam to /opt/containerd/bin/ or /opt/cni/bin

First, we will generate a config:

$> stellar config > stellar.conf

This will produce a default configuration. Edit the addresses to match your environment. For this example we will use the IP 10.0.1.70.

{
    "ConnectionType": "local",
    "ClusterAddress": "10.0.1.70:7946",
    "AdvertiseAddress": "10.0.1.70:7946",
    "Debug": false,
    "NodeID": "dev",
    "GRPCAddress": "10.0.1.70:9000",
    "TLSServerCertificate": "",
    "TLSServerKey": "",
    "TLSClientCertificate": "",
    "TLSClientKey": "",
    "TLSInsecureSkipVerify": false,
    "ContainerdAddr": "/run/containerd/containerd.sock",
    "Namespace": "default",
    "DataDir": "/var/lib/stellar",
    "StateDir": "/run/stellar",
    "Bridge": "stellar0",
    "UpstreamDNSAddr": "8.8.8.8:53",
    "ProxyHTTPPort": 80,
    "ProxyHTTPSPort": 443,
    "ProxyTLSEmail": "",
    "GatewayAddress": "127.0.0.1:9001",
    "EventsAddress": "10.0.1.70:4222",
    "EventsClusterAddress": "10.0.1.70:5222",
    "EventsHTTPAddress": "10.0.1.70:4322",
    "CNIBinPaths": [
        "/opt/containerd/bin",
        "/opt/cni/bin"
    ],
    "Peers": [],
    "Subnet": "172.16.0.0/12"
}

To start the initial node run:

$> stellar -D server --config stellar.conf

To join additional nodes simply add the AdvertiseAddress of the first node to the Peers config option of the second node:

For example:

{
    "ConnectionType": "local",
    "ClusterAddress": "10.0.1.71:7946",
    "AdvertiseAddress": "10.0.1.71:7946",
    "Debug": false,
    "NodeID": "dev",
    "GRPCAddress": "10.0.1.71:9000",
    "TLSServerCertificate": "",
    "TLSServerKey": "",
    "TLSClientCertificate": "",
    "TLSClientKey": "",
    "TLSInsecureSkipVerify": false,
    "ContainerdAddr": "/run/containerd/containerd.sock",
    "Namespace": "default",
    "DataDir": "/var/lib/stellar",
    "StateDir": "/run/stellar",
    "Bridge": "stellar0",
    "UpstreamDNSAddr": "8.8.8.8:53",
    "ProxyHTTPPort": 80,
    "ProxyHTTPSPort": 443,
    "ProxyTLSEmail": "",
    "GatewayAddress": "127.0.0.1:9001",
    "EventsAddress": "10.0.1.71:4222",
    "EventsClusterAddress": "10.0.1.71:5222",
    "EventsHTTPAddress": "10.0.1.71:4322",
    "CNIBinPaths": [
        "/opt/containerd/bin",
        "/opt/cni/bin"
    ],
    "Peers": ["10.0.1.70:7946"],
    "Subnet": "172.16.0.0/12"
}

You will now have a two node cluster. To see node information, use sctl.

$> sctl --addr 10.0.1.70:9000 cluster nodes
NAME                ADDR                OS                       UPTIME              CPUS                MEMORY (USED)
stellar-00          10.0.1.70:9000      Linux (4.17.0-3-amd64)   7 seconds           2                   242 MB / 2.1 GB
stellar-01          10.0.1.71:9000      Linux (4.17.0-3-amd64)   6 seconds           2                   246 MB / 2.1 GB

Deploying an Application

To deploy an application, create an application config. For example, create the following as example.conf:

{
    "name": "example",
    "labels": [
        "env=prod",
        "region=us-east"
    ],
    "services": [
        {
            "name": "redis",
            "image": "docker.io/library/redis:alpine",
            "runtime": "io.containerd.runtime.v1.linux",
            "process": {
                "uid": 0,
                "gid": 0,
                "args": ["redis-server"]
            },
            "labels": [
                "env=prod"
            ],
            "network": true
        }
    ]
}

Then run the following to deploy:

$> sctl --addr 10.0.1.70:9000 apps create -f ./example.conf

You should now see the application deployed:

$> sctl --addr 10.0.1.70:9000 apps list
NAME                SERVICES
example             1

$> sctl --addr 10.0.1.70:9000 apps inspect example
Name: example
Services:
  - Name: example.redis
    Image: docker.io/library/redis:alpine
    Runtime: io.containerd.runtime.v1.linux
    Snapshotter: overlayfs
    Labels:
      containerd.io/restart.status=running
      stellar.io/application=example
      stellar.io/network=true

By default all applications that have networking enabled will have a corresponding nameserver record created. To view the records use the following:

$> sctl --addr 10.0.1.70:9000 nameserver list
NAME                    TYPE                VALUE                                            OPTIONS
example.redis.stellar   A                   172.16.0.4
example.redis.stellar   TXT                 node=stellar-00; updated=2018-09-08T10:71:02-04:00

stellar's People

Contributors

ehazlett avatar hades32 avatar tombee avatar vdemeester avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

stellar's Issues

Bind interface operation not permitted

I am setting up stellar and configuring the first vm. I got this error when trying:

DEBU[0000] starting server agent
DEBU[0000] starting grpc server                          
DEBU[0000] initializing server
DEBU[0000] network init
DEBU[0000] allocating network subnet for node 
DEBU[0000] service.network allocating subnet
DEBU[0000] setting up subnet 172.16.0.0/22
DEBU[0000] setting up local gateway 172.16.0.1
DEBU[0000] setting up gateway
DEBU[0000] bind interface: eth0
FATA[0000] operation not permitted

Skipping local route

At the part where Stellar tried to build the local route and get the error below:

DEBU[0005] services started                              duration=5.0005632s
ERRO[0005] timeout starting service stellar.services.proxy.v1
DEBU[0010] skipping local route 172.16.0.0/22
ERRO[0010] rpc error: code = Unknown desc = failed to dial "/run/containerd/containerd.sock": context deadline exceeded
DEBU[0020] skipping local route 172.16.0.0/22
DEBU[0030] skipping local route 172.16.0.0/22
DEBU[0040] skipping local route 172.16.0.0/22
DEBU[0050] skipping local route 172.16.0.0/22
DEBU[0060] skipping local route 172.16.0.0/22
DEBU[0060] pruning datastore

The last parts of the message just continue until I stop it.

Proxy has unnecessary reloads

There is an issue with the update detection that causes the proxy service to reload often. There is no apparent negative effects as the reloads are cheap but it should be improved. Perhaps switch to a cluster event system for notifications.

Incompatible CNI versions

Hi Evan,

First of all, thank you so much for this project, this is just unbelievably great. I am an ops guys trying to learn go, been researching around containerd and ended-up on your awesome project.

I have installed the latest CNI version following a cillium project doc (https://cilium.readthedocs.io/en/v1.1/kubernetes/install/) as I did not have loopback before.

I thought of raising this issue to ask you what would be the recommended approach, like updating the plugin to use 0.3.1 or for me to downgrade the CNI (couldn't find the right key to download the 0.3.0 version yet) ?

jralmaraz@jralmaraz-vn:~/git/stellar$ sudo sctl --addr 127.0.0.1:9000 apps create -f ./example.conf
FATA[0000] rpc error: code = Unknown desc = incompatible CNI versions; config is "0.3.1", plugin supports ["0.1.0" "0.2.0" "0.3.0"]

Thanks in advance and happy new year !

Cheers,

Jose

makefile package target use case

Hi,
I have a "containerize apps" project and use a makefile to run cobra commands for building app binaries, then an image, then run a container.
Your makefile capability is impressive and I'm keen to adopt some of your design, especially the package target.
What is your intended use case for the package target?

Missing dependency(?)

I wanted to built stellar myself with the built via Docker option (because there are no armhf binaries) and got this

-> building daemon e70a943-dev
../../vendor/github.com/mholt/caddy/caddytls/client.go:30:2: cannot find package "github.com/xenolf/lego/acme" in any of:
        /go/src/github.com/ehazlett/stellar/vendor/github.com/xenolf/lego/acme (vendor tree)
        /usr/local/go/src/github.com/xenolf/lego/acme (from $GOROOT)
        /go/src/github.com/xenolf/lego/acme (from $GOPATH)
make: *** [Makefile:50: daemon] Error 1
 -> Built dev version e70a943 (linux/amd64)

I now manually added a go get ... but now I get

# github.com/ehazlett/stellar/vendor/github.com/mholt/caddy/caddytls
../../vendor/github.com/mholt/caddy/caddytls/config.go:122:10: undefined: acme.KeyType
make: *** [Makefile:50: daemon] Error 2

Race condition in tombstone records

There is a race condition when an app is created, deleted and re-created quickly. There are tombstone records created for the delete which do not get resolved until after the re-create which then remove the valid records in the nameserver service.

Note: due to the eventual consistency this will get resolved on the next datastore replication but can cause issues in the meantime. A possible solution is to check for tombstone records when creating and remove the tombstone to prevent the delete.

macOS

Does this support MacOS ? I would like to use it for Mac desktops in order to run workloads on our Macs :)

It seems contanerd suppors windows via the Windows hyper shim.

Building proto with docker fails

Seems like building is not working via Docker make docker-generate. I'm not familar enough with protoc to know what it means by the plugin failure.

I ran this on two machines to ensure it can be replicated: AWS Instance and my MBP.

[snip]
go: finding github.com/mholt/caddy v0.0.0-20180907212407-13a54dbdda25
 -> building cli 42605bd-dev
 -> building cni-ipam 42605bd-dev
 -> Built dev version 42605bd (linux/amd64)
Removing intermediate container d25e59d175b5
 ---> 55835e7cb1df
Successfully built 55835e7cb1df
Successfully tagged stellar-dev:latest
protoc-gen-grpc-gateway: program not found or is not executable
--grpc-gateway_out: protoc-gen-grpc-gateway: Plugin failed with status code 1.
2018/11/06 17:29:13 protoc -I.:/go/src/github.com/ehazlett/stellar/vendor/github.com/gogo/protobuf:/go/src/github.com/ehazlett/stellar/vendor/github.com/gogo/googleapis:/go/src/github.com/ehazlett/stellar/vendor:/go/src:/usr/local/include:/usr/include --grpc-gateway_out=import_path=github.com/ehazlett/stellar/api/services/application/v1,Mgogoproto/gogo.proto=github.com/gogo/protobuf/gogoproto,Mgoogle/api/annotations.proto=github.com/gogo/googleapis,Mgoogle/api/http.proto=github.com/gogo/googleapis,Mgoogle/protobuf/any.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/descriptor.proto=github.com/gogo/protobuf/protoc-gen-gogo/descriptor,Mgoogle/protobuf/duration.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/empty.proto=github.com/gogo/protobuf/types,Mgoogle/protobuf/timestamp.proto=github.com/gogo/protobuf/types:/go/src /go/src/github.com/ehazlett/stellar/api/services/application/v1/application.proto
make: *** [Makefile:38: generate] Error 123
tar: Unrecognized archive format
tar: Error exit delayed from previous errors.
make: *** [docker-generate] Error 1

Garbage collector

Currently there is no way to clean up containerd automatically just like we would with Docker.
Is it possible to introduce a manual garbage collector for the images, containers, snapshots, content (ctr content) and more (?)?
The command could be like this: sctl gc

Application Networking

In building the network service we need to decide how to handle IP assignment.

  1. Should we assign an IP to every container or just certain ones?
  2. When assigning do we only assign containers and applications that are launched through Stellar or any container in the namespace?

Authentication between nodes

Currently if I want to make a cluster of nodes connected using the internet (not trough a private network like a VPN) anyone can join my cluster because the AdvertiseAddress is exposed on the internet.
It would be great to add an authentication layer with a token or a certificate so that only trusted node can participate in the cluster.

logs management

Currently it's not possible to check the logs of an application (unless we use ctr) launched by stellar, it would be great to add a command like this one to be able to fetch the logs about the application from any node: sctl apps logs <appname>.

endpoints for UDP/TCP

I tried to specify an endpoint for a UDP application like this:

"endpoints": [
    {
        "service": "dns",
        "protocol": "udp",
        "port": 53
    }
]

But it doesn't seem to work. Is the feature implemented or do I've to install Radiant?

Integration Tests

Right now most of the tests are unit based. We need to add integration tests.

Auto update containers

On Nomad and on docker with watchtower it's possible to automatically update the app if a new image from the Docker hub comes up.
I would love to see that feature implemented and maybe an automatic revert to the previous working container if the update fail?

Improve HA for Proxy TLS

Currently when using multiple nodes behind a frontend proxy (i.e. TCP) certificates can take a little while to retrieve as the acme challenge can bounce around to different nodes until it lands on the correct node. There are a couple things we could possibly do:

  • Replicate the certificate storage from a single node
  • Implement a custom TLS storage asset provider for the embedded Caddy

Is there a chat room?

Hi! I'm diving into the code base of stellar in efforts to learn more about containers and go, but in some places that I get stuck, I wonder if there is a chat room I can discuss in? If there is not one currently established, are there any plans to make one?

If wanted I can help with this....

Volumes ?

Hi,

i love the idea of a lightweight orchestrator instead of an overblown kubernetes.
I just created a 2 node cluster within virtual machines to play around if i am able to switch a kubernetes cluster over to stellar....
I also ran over #21 :-D but got the example running.

The mount options is on the same level as docker is using right? Do you have a sample with nfs mounts? How do you handle this stuff?

Thanks Martin!

PS: I gonna prepare a ansible role to setup a node/cluster - i hate doing repeatable tasks manually..

invalid CIDR

Hi,

i tested stellar on 2 VMs before going on baremetal. Everything fine - now on baremetal i get the following error log

$ stellar config --nic enp3s0 > stellar.config
$ stellar -D server --config ./stellar.config
DEBU[0000] seed peers seedPeers="[]"
DEBU[0000] getPeersFromCache: []
DEBU[0000] cluster peers peers="[]" seed_peers="[]"
INFO[0000] registered service id=stellar.services.version.v1
INFO[0000] registered service id=stellar.services.node.v1
INFO[0000] registered service id=stellar.services.health.v1
INFO[0000] registered service id=stellar.services.cluster.v1
INFO[0000] registered service id=stellar.services.datastore.v1
INFO[0000] registered service id=stellar.services.gateway.v1
INFO[0000] registered service id=stellar.services.network.v1
INFO[0000] registered service id=stellar.services.application.v1
INFO[0000] registered service id=stellar.services.nameserver.v1
INFO[0000] registered service id=stellar.services.proxy.v1
INFO[0000] registered service id=stellar.services.events.v1
DEBU[0000] starting server agent
DEBU[0000] starting grpc server addr="172.16.0.6:9000"
DEBU[0000] initializing server
DEBU[0000] network init
DEBU[0000] allocating network subnet for node y
DEBU[0000] service.network allocating subnet
FATA[0000] invalid CIDR address:

The configfile sounds similar:

{
"NodeID": "y",
"GRPCAddress": "172.16.0.6:9000",
"TLSServerCertificate": "",
"TLSServerKey": "",
"TLSClientCertificate": "",
"TLSClientKey": "",
"TLSInsecureSkipVerify": false,
"ContainerdAddr": "/run/containerd/containerd.sock",
"Namespace": "default",
"DataDir": "/var/lib/stellar",
"StateDir": "/run/stellar",
"Bridge": "stellar0",
"UpstreamDNSAddr": "8.8.8.8:53",
"ProxyHTTPPort": 80,
"ProxyHTTPSPort": 443,
"ProxyTLSEmail": "",
"GatewayAddress": "172.16.0.6:9001",
"EventsAddress": "172.16.0.6:4222",
"EventsClusterAddress": "172.16.0.6:5222",
"EventsHTTPAddress": "172.16.0.6:4322",
"CNIBinPaths": [
"/opt/containerd/bin",
"/opt/cni/bin"
],
"ConnectionType": "local",
"ClusterAddress": "172.16.0.6:7946",
"AdvertiseAddress": "172.16.0.6:7946",
"Debug": false,
"Peers": [],
"Subnet": "172.16.0.0/12",
"ProxyHealthcheckInterval": "5s"
}

I tried to modify the subnet, reviewed the code where the output comes from but i cannot get the cause of this failure :-(

Thanks, Martin

failed to find plugin "loopback"

Running into the following error on ARM when attempting to create an app:

sctl --addr 10.0.1.123:9000 apps create -f example.conf
FATA[0000] rpc error: code = Unknown desc = failed to find plugin "loopback" in path [/usr/local/bin /usr/local/bin/cni]

Here's my stellar.conf

{
    "ConnectionType": "local",
    "ClusterAddress": "10.0.1.123:7946",
    "AdvertiseAddress": "10.0.1.123:7946",
    "Debug": false,
    "NodeID": "rpi-cluster-03",
    "GRPCAddress": "10.0.1.123:9000",
    "TLSServerCertificate": "",
    "TLSServerKey": "",
    "TLSClientCertificate": "",
    "TLSClientKey": "",
    "TLSInsecureSkipVerify": false,
    "ContainerdAddr": "/run/containerd/containerd.sock",
    "Namespace": "default",
    "DataDir": "/var/lib/stellar",
    "StateDir": "/run/stellar",
    "Bridge": "stellar0",
    "UpstreamDNSAddr": "10.0.1.1:53",
    "ProxyHTTPPort": 80,
    "ProxyHTTPSPort": 443,
    "ProxyTLSEmail": "",
    "GatewayAddress": "127.0.0.1:9001",
    "EventsAddress": "10.0.1.123:4222",
    "EventsClusterAddress": "10.0.1.123:5222",
    "EventsHTTPAddress": "10.0.1.123:4322",
    "CNIBinPaths": [
        "/usr/local/bin",
        "/usr/local/bin/cni"
    ],
    "Peers": [],
    "Subnet": "172.16.0.0/12",
	"ProxyHealthcheckInterval": "5s"
}

The example config's CNIBinPaths appeared to specify containerd and cni binaries. I grabbed the latest release of cni for ARM and your ARM build of containerd. Here's my /usr/local/bin:

|-- cni
|   |-- cnitool
|   `-- noop
|-- containerd
|-- containerd-shim
|-- containerd-shim-runc-v1
|-- containerd-shim-runc-v2
|-- ctr
|-- sctl
|-- stellar
`-- stellar-cni-ipam

Am I missing a cni plugin for this to work? Thanks for all of your help so far!

Proxy service configured with duplicates

Occasionally the proxy service can report duplicate upstreams. The nameserver service appears to have the correct data so this looks like a bug in the local query to get applications:

Proxy

foo.com              http://172.16.8.4:80 (up: 479.478µs), http://172.16.0.4:80 (up: 947.151µs), http://172.16.4.5:80 (up: 826.847µs), http://172.16.8.4:80 (up: 2.05114ms)

Nameserver Records

blog.app00.stellar       A                   172.16.0.4
blog.app00.stellar       TXT                 node=ctr-00; updated=2018-10-24T23:42:16-04:00
blog.app00.stellar       SRV                 blog.app00.stellar                               service=web proto=http priority=0 weight=0 port=80
blog.app01.stellar       A                   172.16.8.4
blog.app01.stellar       TXT                 node=ctr-02; updated=2018-10-24T23:42:21-04:00
blog.app01.stellar       SRV                 blog.app01.stellar                               service=web proto=http priority=0 weight=0 port=80
blog.app02.stellar       A                   172.16.4.5
blog.app02.stellar       TXT                 node=ctr-01; updated=2018-10-24T23:42:28-04:00
blog.app02.stellar       SRV                 blog.app02.stellar                               service=web proto=http priority=0 weight=0 port=80

How do I use the gRPC Gateway API?

I found here #8 that it's possible to interact with the API using HTTP but I don't really know the endpoints of that API, is it possible to help me how to find them or maybe add swagger for an easy use of the API?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.