tsuru/bs

Big siblings watch their siblings and report to big tsuru parent.

tsuru

What is tsuru?

tsuru is an extensible and open source Platform as a Service (PaaS) that makes application deployments faster and easier. With tsuru, you don’t need to think about servers at all. As an application developer, you can:

• Write apps in the programming language of your choice
• Back apps with add-on resources such as SQL and NoSQL databases, including memcached, Redis, and many others
• Manage apps using the tsuru command-line tool

Links:

• Landing page: https://tsuru.io
• Full Documentation: https://docs.tsuru.io/main/
• How to Contribute: https://docs.tsuru.io/stable/contributing/
• Repository & Issue Tracker: https://github.com/tsuru/tsuru
• Talk to us on Gitter: https://gitter.im/tsuru/tsuru

Popular platforms supported:

• Python
• Nodejs
• GO
• Ruby
• PHP
• Perl
• Lua
• Java

Quick Start

Getting tsuru-client

Download the latest release for your platform at: https://github.com/tsuru/tsuru-client/releases/

Example for release 1.1.1 and OS X:

$ curl -sSL https://github.com/tsuru/tsuru-client/releases/download/1.1.1/tsuru-1.1.1-darwin_amd64.tar.gz | tar xz

Install Guides

• Minikube
• GKE - Google Kubernetes Engine

Testing

If everything's gone well you have the tsuru running in a Kubernetes Cluster. Call app list to see tsuru working, this command needs to return one app called tsuru-dashboard.

$ tsuru app list

Local development

Dependencies

Before starting, make sure you have the following tools installed:

• docker (or podman)
• minikube
• go
• yq

You'll also need the Tsuru Client to interact with the Tsuru API. If you haven't installed it yet, please do so.

For macOS users: We recommend using the qemu driver with socket_vmnet for Minikube clusters. For more information on installing qemu and socket_vmnet, refer to the following links:

• qemu
• socket_vmnet

Note: If you are using Docker-compatible alternatives like Podman, be sure to specify the DOCKER variable with the correct binary when running make commands. For example: make local.run DOCKER=podman.

Running local environment

To run the Tsuru API locally, you'll need to first set up the local environment. This setup process is crucial because it creates the default configuration files, initializes required dependencies, and prepares your local system to host the Tsuru API. The following command will handle all these tasks:

make local.setup

Once the setup is complete, you won’t need to run this command again unless you want to reset your environment.

After the initial setup, you can start the Tsuru API and its dependencies using the following command:

make local.run

Once the Tsuru API is running, open a new terminal window and configure your Tsuru CLI to point to the local-dev target. This target tells the CLI to interact with your local Tsuru API instance rather than a remote server. You can set the target using this command:

tsuru target-set local-dev

Tsuru's targets function similarly to Kubernetes' kubectl config contexts, allowing you to switch between different environments easily.

To confirm that everything is set up correctly, you can log in and list the clusters managed by your Tsuru API instance:

tsuru login [email protected] # password: admin@123
tsuru cluster list

If everything is working as expected, you should see your local Minikube cluster listed as the default provisioner.

Cleaning up

When you're done working with your local environment, it's important to stop the services to free up system resources. You can stop the dependencies using:

make local.stop

If you want to fully reset your environment, or if you no longer need the Tsuru API and its dependencies on your local machine, you can remove all associated resources using:

make local.cleanup

bs's Issues

Metrics for incoming connections

Currently only metrics for incoming connections are being tracked.
This happens due to https://github.com/tsuru/bs/blob/master/metric/conntrack.go#L52 only reading IP/Ports from Metas[0] which tracks original connection IPs. If we want also to add metrics for incoming connections we need to read Metas[1] which tracks reply IPs so we can match the src reply IP with the container IP.

Collect containers block io metrics

Docker now exposes container metrics on block IO (read and write bytes) on the stats api. It would be interesting to have bs collect and report these metrics to the metrics backend.

Report metrics from non-apps containers

Bs should be able to report metrics from other containers running on the docker host, including itself.

This can be accomplished by setting an env variable that will contain the env variable name that needs to be present on containers that are going to have metrics collected. The default behaviour, when the env variable is not set, will be to collect metrics from all containers.

support metrics

bs should collect metrics on application containers and send them to the metrics provider configured in the application, as tsuru-circus used to do before lean containers (tsuru/tsuru#1136).

GELF protocol support

I think that protocol may be a good alternative for structured logs.

http://docs.graylog.org/en/2.2/pages/gelf.html

Host system wide metrics

bs should collect and send system wide metrics(load, mem, cpu etc) about its host.

Support Prometheus as metrics backend

Prometheus is one of the new metrics/monitoring/alerting system, has been developed by Soundcloud. This system has gained a lot of traction and popularity, and it's being used in some of the big companies.

Would be nice to add support for prometheus.

implement rsyslog multiplexing

bs should listen for the rsyslog protocol and multiple it to other log providers, like the tsuru API and an actual remote syslog server. So, we would use the syslog log driver in Docker, pointing to the container running bs, and the bs would send to where it's configured to run (a rsyslog server, the tsuru logging API, a logplex server, wherever).

syslog forwarding not working

Hi

I had configure month ago a syslog forwarder with syslog but now i can't get any tsuru app log on remote syslog

I had reconfigure bs with the new LOG_XXX variables but nothing. Last i had activate debug on bs and had see this error on docker logs

2016/05/17 11:06:59 [DEBUG] [log forwarder] ignored msg syslogparser.LogParts{"client":"[::1]:47604", "hostname":"docker/af33eefd8c90[2326]:", "tag":"", "facility":3, "rawmsg":[]uint8{0x3c, 0x32, 0x37, 0x3e, 0x4d, 0x61, 0x79, 0x20, 0x31, 0x37, 0x20, 0x31, 0x33, 0x3a, 0x30, 0x36, 0x3a, 0x35, 0x39, 0x20, 0x64, 0x6f, 0x63, 0x6b, 0x65, 0x72, 0x2f, 0x61, 0x66, 0x33, 0x33, 0x65, 0x65, 0x66, 0x64, 0x38, 0x63, 0x39, 0x30, 0x5b, 0x32, 0x33, 0x32, 0x36, 0x5d, 0x3a, 0x20, 0x5b, 0x70, 0x69, 0x64, 0x3a, 0x20, 0x39, 0x7c, 0x61, 0x70, 0x70, 0x3a, 0x20, 0x30, 0x7c, 0x72, 0x65, 0x71, 0x3a, 0x20, 0x32, 0x35, 0x37, 0x33, 0x37, 0x35, 0x2f, 0x32, 0x35, 0x37, 0x33, 0x37, 0x35, 0x5d, 0x20, 0x31, 0x37, 0x32, 0x2e, 0x31, 0x36, 0x2e, 0x37, 0x30, 0x2e, 0x31, 0x31, 0x34, 0x20, 0x28, 0x29, 0x20, 0x7b, 0x34, 0x38, 0x20, 0x76, 0x61, 0x72, 0x73, 0x20, 0x69, 0x6e, 0x20, 0x39, 0x32, 0x38, 0x20, 0x62, 0x79, 0x74, 0x65, 0x73, 0x7d, 0x20, 0x5b, 0x54, 0x75, 0x65, 0x20, 0x4d, 0x61, 0x79, 0x20, 0x31, 0x37, 0x20, 0x31, 0x33, 0x3a, 0x30, 0x36, 0x3a, 0x35, 0x39, 0x20, 0x32, 0x30, 0x31, 0x36, 0x5d, 0x20, 0x47, 0x45, 0x54, 0x20, 0x2f, 0x6c, 0x6f, 0x67, 0x2f, 0x70, 0x72, 0x6f, 0x64, 0x75, 0x63, 0x74, 0x2f, 0x30, 0x33, 0x35, 0x31, 0x33, 0x32, 0x34, 0x38, 0x35, 0x3f, 0x5f, 0x3d, 0x35, 0x33, 0x35, 0x61, 0x31, 0x65, 0x33, 0x38, 0x64, 0x39, 0x26, 0x76, 0x3d, 0x73, 0x77, 0x5f, 0x70, 0x72, 0x69, 0x6e, 0x74, 0x26, 0x68, 0x3d, 0x31, 0x38, 0x65, 0x32, 0x35, 0x36, 0x64, 0x32, 0x37, 0x62, 0x39, 0x33, 0x34, 0x39, 0x66, 0x31, 0x30, 0x62, 0x35, 0x30, 0x61, 0x64, 0x34, 0x62, 0x34, 0x61, 0x34, 0x39, 0x66, 0x62, 0x32, 0x65, 0x30, 0x65, 0x34, 0x37, 0x62, 0x31, 0x63, 0x61, 0x20, 0x3d, 0x3e, 0x20, 0x67, 0x65, 0x6e, 0x65, 0x72, 0x61, 0x74, 0x65, 0x64, 0x20, 0x34, 0x33, 0x20, 0x62, 0x79, 0x74, 0x65, 0x73, 0x20, 0x69, 0x6e, 0x20, 0x38, 0x20, 0x6d, 0x73, 0x65, 0x63, 0x73, 0x20, 0x28, 0x48, 0x54, 0x54, 0x50, 0x2f, 0x31, 0x2e, 0x31, 0x20, 0x32, 0x30, 0x30, 0x29, 0x20, 0x37, 0x20, 0x68, 0x65, 0x61, 0x64, 0x65, 0x72, 0x73, 0x20, 0x69, 0x6e, 0x20, 0x32, 0x35, 0x37, 0x20, 0x62, 0x79, 0x74, 0x65, 0x73, 0x20, 0x28, 0x31, 0x20, 0x73, 0x77, 0x69, 0x74, 0x63, 0x68, 0x65, 0x73, 0x20, 0x6f, 0x6e, 0x20, 0x63, 0x6f, 0x72, 0x65, 0x20, 0x30, 0x29}, "timestamp":time.Time{sec:63599087219, nsec:0, loc:(*time.Location)(0xf3a360)}, "content":"9|app: 0|req: 257375/257375] 172.16.70.114 () {48 vars in 928 bytes} [Tue May 17 13:06:59 2016] GET /log/product/035132485?_=535a1e38d9&v=sw_print&h=18e256d27b9349f10b50ad4b4a49fb2e0e47b1ca => generated 43 bytes in 8 msecs (HTTP/1.1 200) 7 headers in 257 bytes (1 switches on core 0)", "priority":27, "severity":3} error to get appname: No such container: docker/af33eefd8c90[2326]:

Every error is about can't get appname. This is my bs env:

root@tsrmgr:~# tsuru-admin bs-info 
Image: 

Environment Variables [Default]:
+------------------------------+-------------------------+
| Name                         | Value                   |
+------------------------------+-------------------------+
| SYSLOG_FORWARD_ADDRESSES     | udp://172.16.70.66:5140 |
| STATUS_INTERVAL              | 60                      |
| METRICS_INTERVAL             | 60                      |
| METRICS_BACKEND              | logstash                |
| LOG_SYSLOG_FORWARD_ADDRESSES | udp://172.16.70.66:5140 |
| LOG_SYSLOG_TIMEZONE          | Europe/Madrid           |
+------------------------------+-------------------------+

root@tsr01:~# docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

failed to load system roots and no roots provided

Today we upgrade an environment which restart the bs process, and now we are hitting this kind of error:

2015/11/09 17:49:30 [WARNING] invalid interval "". Using the default value of 60 seconds
2015/11/09 17:49:30 [WARNING] invalid metrics interval "". Using the default value of 60 seconds
2015/11/09 17:49:30 Unable to initialize log forwarder: x509: failed to load system roots and no roots provided

If you something this, it fixes the issue:

docker run -e "DOCKER_ENDPOINT=http://1.2.3.4:4243" -e "TSURU_ENDPOINT=https://api.example.com:443/" -e "TSURU_TOKEN=fffffffffff" -e "SYSLOG_LISTEN_ADDRESS=udp://0.0.0.0:1514" -e "PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"  --entrypoint "bash" tsuru/bs:v1 -c 'apt-get update && apt-get install -y ca-certificates && /bin/bs'

I think we need to install the ca-certificates in the image.

Also, maybe we should pin the versions

use gops to report informations about the bs process

gops (https://github.com/google/gops) is an agent that can report additional information about the Go process. We can use it to make easier to debug problems in bs

Support read logs by docker API

docker supports to define a application to implement LogDriver protocol, this is useful to create one possibility to read logs directly by kubectl logs.

documentation: https://docs.docker.com/engine/extend/plugins_logging/
log driver example: https://github.com/cpuguy83/docker-log-driver-test
FIFO: https://en.wikipedia.org/wiki/Named_pipe

Statsd always sends 0 as metric value

Currently statsd statter ignores the value passed to the function and always sends 0 as metric value: https://github.com/tsuru/bs/blob/master/metric/statsd.go#L54

GELF messages are not being sent with timestamp

The implementation of the GELF messages does not include the timestamp telling when the message was created, this can lead to misinformation in scenarios where the message is not processed immediately.

Code:

bs/log/gelfforwarder.go

Line 64 in 649c333

msg := &gelf.Message{

I believe the field we are missing this field:
https://github.com/Graylog2/go-gelf/blob/v1/gelf/writer.go#L54