silenteh/gantryos

GantryOS: distributed systems kernel

gantryos's Issues

[MASTER] allow data write to slaves

The master needs to keep a map of all the slave connections and it needs to be able to write to them back in order to start tasks.
Also it needs to make sure it removes the connections with the slaves which are no longer valid from the map.

We need extensive tests and integration tests for this

[PROXY] We need to support several features

• different load balancing
• ACLs
• rules ordering
• rewrite rules
• Stats

Docker task: events

Track events of docker container

https://docs.docker.com/reference/api/docker_remote_api_v1.17/#monitor-dockers-events

[LOGGING] The master should run the syslog

Although configurable from the config, the master should run the syslog server, so we lower the entry barriers for users.
We pre-configure it

• Allow via config to enable syslog on the master
• Preinstall syslog server well configured (syslog-ng most probably on udp)
• Send the info to the master syslog from the slaves (each container)

[TASK] Add XEN

It would be really great if we could add XEN to gantry so users can spin up VMs

[ALERTING] Integrate with alerting systems

• Integrate with pager duty

[Docker task] - tap into logging of each container

We need to read the logs of each container and send them either to a syslog server or to gantryos logs

[Docker task] - create tests and integration tests

We need to test that the config and host config are filled in correctly

[Task] add LXC

[METRICS] All data containers and VMs metrics are sent to the master

• Read info about the each slave (CPU, Mem, disk space, I/O, network)
• Send the info to the master every 15 seconds (fixed for now)
• Store the information in something like influxdb
• Display the information on the UI

[MONITORING] Read cgroups information and aggregate them to a central place

• Read info about the cgroups on each slave
• Send the info to the master every 15 seconds (fixed for now)
• Store the information in something like influxdb
• Display the information on the UI

[TASK] add persistance layer

Leveldb seems to be a good choice

[TASK] add a cron to clean up images and exited containers

In case a container died we need to clean them up with the command:

docker rm $(docker ps -aq)

[TASK] add health check

When the container starts check if the health check passes if not send a signal and take down the container.
If not health check is specified then skip the health check

[LICENSE] Check the license of all the libs and techs we are using

We need to know if we can embed them in a closed source software, or if we need to open source it

[TASK] add container name and app name info

We need to store somewhere a correlation between container id, container name (which can be passed by the user) and the app name and version that the container belongs to

[UI] Create a UI interface which leverages the REST

We need a UI interface to execute the main tasks

[GANTRYOS] Maintenance mode

Allow a slave to be put in maintenance mode.
This means that all tasks are relocated somewhere else because the slave needs to be decomisioned or serviced

[Docker task] - add cpu and ram set

We need to add them to the config so that cgroups can assign specific amout of cpu and ram

[GANTRYOS] add the groups concept

[SLAVE REGISTRATION] handle the flag registered on the slave.json

[GANTRYOS] Add the application and deployment concept

We need to be able to add an application which consists of one or more tasks
In addition we need to add the deployment concept

[MASTER] Enable clustering for high availability

We need to find a way to replicate the application status from the master0 to the other standby masters.
This is helpful also in case the master need to be serviced or upgraded

[REST] Create a rest interface for executing tasks and managing gantryos

We need a rest interface foe gantry os.
The rest interface will be used also from the UI

[TASK] cleanup after a task is done

[OVS] add basic openflow implemenation

We need to be able to act on the switch data path (flow) to block specific stuff or to add specific flows

• Allow to add basic data flows on both directions

[OVS] add OVSDB implementation

We need to be able to create a switch (bridge) and to add additional stuff like QoS and VLANs and GRE

• Add OVSDB client with connection handling. Need an RCP JSON client.
• Client can create bridges + ports + interfaces
• Client can delete bridges + ports + interfaces
• Client can set VLANs (VPCs)
• Client can set Docker interfaces via micro-container
• Client keeps a list of allocated interfaces

[Docker task] - add volume deletion

There is the possibility of completely remove the data when the container is stopped

[TASK] unify taskstatus messages

We need to write consistent status messages.
At the moment I have this mess:

Starting new container with image: google/golang-hello
Pulling the image google/golang-hello
create
start
Container from image google/golang-hello started successfully
Container id5224e3767ce905a2d474da4c67e7accc5f93e4163769ac0917c4dc2a05aacb00 RUNNING
die
stop

[DISCOVERY] add service discovery

allow each container to be discovered via etcd and consul

• Add Consul
  • Add service
  • Add Health check
• Add Etcd

[TASK] Add KVM

It would be really great if we could add KVM to gantry so users can spin up VMs

[PROXY] Integrate HAProxy to load balance apps

Apps should be load balanced by HAProxy which receives events from the master about the deployed app