qubitproducts / bamboo Goto Github PK
View Code? Open in Web Editor NEWHAProxy auto configuration and auto service discovery for Mesos Marathon
License: Apache License 2.0
HAProxy auto configuration and auto service discovery for Mesos Marathon
License: Apache License 2.0
In my use case i use bamboo for update HAproxy and for Redis failover.
Today i need to have one instance of Bamboo for HAproxy and one instance of Bamboo for redis failover.
It will be nice if i can use only once instance of Bamboo to udpate HAproxy and Redis failover.
Hi,
im new to go so maybe just a simple error on my side.
Downloaded all the git stated in bamboo.go (zookeeper etc.) and did go bamboo.go which results in:
# command-line-arguments
./bamboo.go:119: unknown lumberjack.Logger field 'Filename' in struct literal
Best,
Mike
Currently, the mode needs to be hardcoded e.g. "mode tcp" or "mode http". Can we make it configurable by adding tags for a given service so that we can use "$services.mode" to get the value ?
We can probably store key/value map to make it generic e.g. $services.param["mode"] to make any key referenced in ha proxy template.
When I scale my app up and there are 4 of 10 apps running bamboo creates config for all of them.
Currently I do not have health checks (that is how I've found out about this) and I am aware I should have those implemented.
IMO this could be handled by bamboo which would create haproxy configuration only for the apps which are currenly up.
Is this info available from marathon?
Is marathon sending callbacks about another app up?
Hi,
I'm attempting to deploy bamboo in a docker container as described in the documentation. I've built the docker image, but it is unable to find the config file when I run it. Here's what I'm seeing:
[root@bamboo bamboo-0.2.9]# ls -l config/bamboo_test.json
-rw-r--r--. 1 root root 642 Mar 6 20:51 config/bamboo_test.json
[root@bamboo bamboo-0.2.9]# docker run -t -i --rm -p 8000:8000 -p 80:80 \
-e MARATHON_ENDPOINT=http://xxx.xxx.xxx.xxx:8080 \
-e BAMBOO_ENDPOINT=http://xxx.xxx.xxx.xxx:8000 \
-e BAMBOO_ZK_HOST=xxx.xxx.xxx.xxx:2181 \
-e BAMBOO_ZK_PATH=/bamboo \
-e BIND=":8000" \
-e CONFIG_PATH="config/bamboo_test.json" \
-e BAMBOO_DOCKER_AUTO_HOST=true \
bamboo
sed: can't read config/bamboo_test.json: No such file or directory
Thanks in advance for the help.
Using Mesos 0.20.1 and Marathon 0.7.0 I commonly see multiple HAProxy instances after a deployment event. It looks like Marathon doesn't batch events so Bamboo ends up restarting HAProxy multiple times in rapid succession. This leads to HAProxy not being able properly reload and thus multiple instances.
Every reload cmd leaves the previous haproxy process as zombie. Because neither bamboo nor docker waits for them, they don't go away.
In a real unix system (if bamboo is installed as package) init is waiting for the exited processes and this doesn't happen.
Hi,
here is where I am with adding supervisord to docker.
Currently I am just triggering bamboo bin and I suppose it runs haproxy.
Is there a way we could run haproxy separately as another target in supervisord.conf?
The README about the REST API says that marathon application ids have to be URI encoded. The given examples in the documentation are actually double encoded. I’ve tried both, single encoded and double encoded marathon app ids, the REST API only works with double encoded ids.
curl -i -X DELETE http://localhost:8000/api/services/%252Fapp-1
http://localhost:8000/api/services/%2Fapp-1
http://localhost:8000/api/services//app-1
Currently, to get a "create or update" behaviour, clients have to implement it themselves, because POSTing with an existing id will fail, as will putting with a non-existing id. It would be convenient if PUT could be used for creating new configurations too.
I have successful deploy the bamboo, the http instance are works like a charm. so i have a another request, how can i config https service?
Load balancing of TCP services using haproxy
This avoids TTL for some cases when DNS provider is less helpful.
Also a UI element to show if HAProxy is running on localhost (green == process started) ?
Hi,
any specific reason for the colons in
EscapedId: strings.Replace(appId, "/", "::", -1),
?
The colons mess up my host acl
acl ::socketchat-host hdr(host) -i ::socketchat.example.io
Best,
Mike
Hello.
I'm testing bamboo on a 1 master/3 slaves mesos cluster. Marathon, zk, mesos master, haproxy and bamboo are all on master server.
When I scale up or down a app in Marathon, I have to clic on "update" button in bamboo ui for bamboo to update haproxy configuration. The number of instances is reflected correctly, but the log does not show anything, and haproxy is not reconfigured.
Here is the production.json:
{
"Marathon": {
"Endpoint": "http://localhost:8080",
"Zookeeper": {
"Host": "localhost:2181",
"Path": "/marathon/state",
"ReportingDelay": 5
}
},
"Bamboo": {
"Endpoint": "http://localhost:8000",
"Zookeeper": {
"Host": "localhost",
"Path": "/marathon-haproxy/state",
"ReportingDelay": 5
}
},
"HAProxy": {
"TemplatePath": "/root/gopath/src/github.com/QubitProducts/bamboo/config/haproxy_template.cfg",
"OutputPath": "/usr/local/etc/haproxy/haproxy.cfg",
"ReloadCommand": "read PIDS < /var/run/haproxy.pid; haproxy -f /usr/local/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done"
},
"StatsD": {
"Enabled": false,
"Host": "localhost:8125",
"Prefix": "bamboo-server.development."
}
}
Is there any other configuration to debug te (marathon | zk) /bamboo communication?
Thanks.
The haproxy reload script frequently locks up, prevent config updates for haproxy. I found two related commits: 3bf7fd5 d1d3b65
Note how the second one changes from reading the PID file to using pidof
. On our server where the reload hangs, pidof
reports two running haproxy processes, while there is only one PID in the file - which seems to be the issue mentioned by @sttts in the first commit. The existence of the extra PID, in combination with the while
loop from the first commit causes updates to hang: While haproxy reloads sucessfully, the while
will never terminate, so further updates will not trigger reloads anymore.
$ cat /var/run/haproxy.pid
25296
$ pidof haproxy
25296 22555
$ ps aux|grep haproxy
haproxy 22555 0.0 0.1 30512 3072 ? Ss 09:27 0:01 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 22492
1000 22707 0.0 0.0 9356 644 pts/2 R+ 13:57 0:00 grep --color=auto haproxy
root 25293 0.3 0.0 4400 636 ? S 10:34 0:41 sh -c PIDS=`pidof haproxy`; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done
haproxy 25296 0.0 0.1 30696 3072 ? Ss 10:34 0:02 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 22555
I wonder why the production.example.json was updated in that second commit (cc @activars): to me, it seems that reading the PIDs from the PID file is the correct way.
It's better to keep a local copy of the dependency source code in Godep.
this ensure the build process is consistent and independent of the remote changes.
Hi,
haproxy reloads every 30 seconds even if there are no changes in marathon.
2014/09/11 13:24:45 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:24:50.935589 Marathon: State changed
2014/09/11 13:24:51.104599 HAProxy: Configuration updated
2014/09/11 13:24:51.142967 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
2014/09/11 13:25:15 qzk.go:51: Got event. Delaying post
2014/09/11 13:25:15 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:25:20.936585 Marathon: State changed
2014/09/11 13:25:21.102402 HAProxy: Configuration updated
2014/09/11 13:25:21.135208 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
2014/09/11 13:25:45 qzk.go:51: Got event. Delaying post
2014/09/11 13:25:45 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:25:50.933389 Marathon: State changed
2014/09/11 13:25:51.095557 HAProxy: Configuration updated
2014/09/11 13:25:51.132177 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
2014/09/11 13:26:15 qzk.go:51: Got event. Delaying post
2014/09/11 13:26:15 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:26:20.933709 Marathon: State changed
2014/09/11 13:26:21.107363 HAProxy: Configuration updated
2014/09/11 13:26:21.144372 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
Best,
Mike
I run the latest (as of today) bamboo version and run into the following issue:
config:
{
"Marathon": {
"Endpoint": "http://localhost:8080"
},
"Bamboo": {
"Endpoint": "http://10.218.146.232:8000",
"Zookeeper": {
"Host": "10.218.146.232:2181",
"Path": "/marathon-haproxy/state",
"ReportingDelay": 5
}
},
"HAProxy": {
"TemplatePath": "/opt/bamboo/config/haproxy_template.cfg",
"OutputPath": "/etc/haproxy/haproxy.cfg",
"ReloadCommand": "PIDS=`pidof haproxy`; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done"
},
"StatsD": {
"Enabled": false,
"Host": "localhost:8125",
"Prefix": "bamboo-server.development."
}
}
I deployed bamboo on ubuntu 14.04 (master) and created a new app:
marathon Id: "jenkins"
domain: "jenkins.rnatest.brightcove.com"
Nothing seems to update in the UI and looking a the logs i see:
2014/09/09 21:11:04.189694 [ip-10-146-205-81/tF2Zg1BkAD-000001] Started POST "/api/state/domains" from 50.197.95.1:34552
2014/09/09 21:11:04.192254 [ip-10-146-205-81/tF2Zg1BkAD-000001] Returning 200 in 2.499162ms
2014/09/09 21:11:04.295404 [ip-10-146-205-81/tF2Zg1BkAD-000002] Started GET "/api/state" from 50.197.95.1:34552
2014/09/09 21:11:04.438776 [ip-10-146-205-81/tF2Zg1BkAD-000002] panic: runtime error: index out of range
2014/09/09 21:11:04.439196 [ip-10-146-205-81/tF2Zg1BkAD-000002] Returning 500 in 143.765582ms
2014/09/09 21:11:06.030963 [ip-10-146-205-81/tF2Zg1BkAD-000003] Started GET "/" from 50.197.95.1:34552
2014/09/09 21:11:06.031108 [ip-10-146-205-81/tF2Zg1BkAD-000003] Returning 304 in 95.871us
2014/09/09 21:11:06.231419 [ip-10-146-205-81/tF2Zg1BkAD-000004] Started GET "/dist/main-app.js" from 50.197.95.1:52581
2014/09/09 21:11:06.231478 [ip-10-146-205-81/tF2Zg1BkAD-000004] Returning 304 in 31.071us
2014/09/09 21:11:06.231700 [ip-10-146-205-81/tF2Zg1BkAD-000005] Started GET "/dist/main.css" from 50.197.95.1:59688
2014/09/09 21:11:06.231743 [ip-10-146-205-81/tF2Zg1BkAD-000005] Returning 304 in 22.723us
2014/09/09 21:11:06.231938 [ip-10-146-205-81/tF2Zg1BkAD-000006] Started GET "/dist/main-libs.js" from 50.197.95.1:34552
2014/09/09 21:11:06.231980 [ip-10-146-205-81/tF2Zg1BkAD-000006] Returning 304 in 21.994us
2014/09/09 21:11:06.490896 [ip-10-146-205-81/tF2Zg1BkAD-000007] Started GET "/api/state" from 50.197.95.1:59688
2014/09/09 21:11:06.642818 [ip-10-146-205-81/tF2Zg1BkAD-000007] panic: runtime error: index out of range
2014/09/09 21:11:06.643170 [ip-10-146-205-81/tF2Zg1BkAD-000007] Returning 500 in 152.238341ms
2014/09/09 21:11:07.015312 [ip-10-146-205-81/tF2Zg1BkAD-000008] Started GET "/" from 50.197.95.1:59688
2014/09/09 21:11:07.017793 [ip-10-146-205-81/tF2Zg1BkAD-000008] Returning 200 in 2.421628ms
2014/09/09 21:11:07.189494 [ip-10-146-205-81/tF2Zg1BkAD-000009] Started GET "/dist/main-libs.js" from 50.197.95.1:59688
2014/09/09 21:11:07.195451 [ip-10-146-205-81/tF2Zg1BkAD-000010] Started GET "/dist/main-app.js" from 50.197.95.1:52581
2014/09/09 21:11:07.195998 [ip-10-146-205-81/tF2Zg1BkAD-000011] Started GET "/dist/main.css" from 50.197.95.1:34552
2014/09/09 21:11:07.297142 [ip-10-146-205-81/tF2Zg1BkAD-000010] Returning 200 in 101.650699ms
2014/09/09 21:11:07.297552 [ip-10-146-205-81/tF2Zg1BkAD-000011] Returning 200 in 101.5153ms
2014/09/09 21:11:07.863142 [ip-10-146-205-81/tF2Zg1BkAD-000009] Returning 200 in 673.584726ms
2014/09/09 21:11:08.155449 [ip-10-146-205-81/tF2Zg1BkAD-000012] Started GET "/fonts/ionicons.ttf?v=1.5.2" from 50.197.95.1:52581
2014/09/09 21:11:08.155677 [ip-10-146-205-81/tF2Zg1BkAD-000013] Started GET "/api/state" from 50.197.95.1:59688
2014/09/09 21:11:08.156843 Marathon: State changed
2014/09/09 21:11:08.545464 [ip-10-146-205-81/tF2Zg1BkAD-000012] Returning 200 in 389.94999ms
2014/09/09 21:11:08.742842 [ip-10-146-205-81/tF2Zg1BkAD-000014] Started GET "/favicon.ico" from 50.197.95.1:52581
2014/09/09 21:11:08.742926 [ip-10-146-205-81/tF2Zg1BkAD-000014] Returning 404 in 41.233us
2014/09/09 21:11:08.776081 [ip-10-146-205-81/tF2Zg1BkAD-000013] panic: runtime error: index out of range
2014/09/09 21:11:08.776480 [ip-10-146-205-81/tF2Zg1BkAD-000013] Returning 500 in 620.766409ms
2014/09/09 21:11:08.834330 Starting Goji on [::]:8000
2014/09/09 21:11:10.854234 [ip-10-146-205-81/ZJyf9oeUzk-000001] Started GET "/api/state/domains" from 127.0.0.1:50246
2014/09/09 21:11:10.855212 [ip-10-146-205-81/ZJyf9oeUzk-000001] Returning 200 in 930.746us
2014/09/09 21:11:38.142537 Marathon: State changed
2014/09/09 21:11:38.472626 Starting Goji on [::]:8000
2014/09/09 21:12:08.153508 Marathon: State changed
2014/09/09 21:12:08.424940 Starting Goji on [::]:8000
2014/09/09 21:12:38.143537 Marathon: State changed
2014/09/09 21:12:38.471640 Starting Goji on [::]:8000
2014/09/09 21:13:08.636313 Marathon: State changed
2014/09/09 21:13:08.951384 Starting Goji on [::]:8000
2014/09/09 21:13:38.144468 Marathon: State changed
2014/09/09 21:13:38.418398 Starting Goji on [::]:8000
2014/09/09 21:14:08.143968 Marathon: State changed
2014/09/09 21:14:08.447555 Starting Goji on [::]:8000
2014/09/09 21:14:36.398245 [ip-10-146-205-81/I7hhxpqesc-000001] Started GET "/" from 50.197.95.1:13641
2014/09/09 21:14:36.401455 [ip-10-146-205-81/I7hhxpqesc-000001] Returning 200 in 3.141018ms
2014/09/09 21:14:36.645516 [ip-10-146-205-81/I7hhxpqesc-000002] Started GET "/dist/main-libs.js" from 50.197.95.1:13641
2014/09/09 21:14:36.645762 [ip-10-146-205-81/I7hhxpqesc-000003] Started GET "/dist/main-app.js" from 50.197.95.1:60232
2014/09/09 21:14:36.649878 [ip-10-146-205-81/I7hhxpqesc-000004] Started GET "/dist/main.css" from 50.197.95.1:24151
2014/09/09 21:14:36.832260 [ip-10-146-205-81/I7hhxpqesc-000003] Returning 200 in 186.448961ms
2014/09/09 21:14:36.832417 [ip-10-146-205-81/I7hhxpqesc-000004] Returning 200 in 182.492761ms
2014/09/09 21:14:37.412150 [ip-10-146-205-81/I7hhxpqesc-000002] Returning 200 in 766.568162ms
2014/09/09 21:14:37.759763 [ip-10-146-205-81/I7hhxpqesc-000005] Started GET "/api/state" from 50.197.95.1:13641
2014/09/09 21:14:37.763757 [ip-10-146-205-81/I7hhxpqesc-000006] Started GET "/fonts/ionicons.ttf?v=1.5.2" from 50.197.95.1:24151
2014/09/09 21:14:37.902539 [ip-10-146-205-81/I7hhxpqesc-000005] panic: runtime error: index out of range
2014/09/09 21:14:37.903141 [ip-10-146-205-81/I7hhxpqesc-000005] Returning 500 in 143.334318ms
2014/09/09 21:14:38.154143 [ip-10-146-205-81/I7hhxpqesc-000006] Returning 200 in 390.338903ms
2014/09/09 21:14:38.438154 [ip-10-146-205-81/I7hhxpqesc-000007] Started GET "/favicon.ico" from 50.197.95.1:24151
2014/09/09 21:14:38.438222 [ip-10-146-205-81/I7hhxpqesc-000007] Returning 404 in 31.211us
2014/09/09 21:14:38.601105 Marathon: State changed
2014/09/09 21:14:38.892008 Starting Goji on [::]:8000
2014/09/09 21:15:08.148493 Marathon: State changed
2014/09/09 21:15:08.416597 Starting Goji on [::]:8000
2014/09/09 21:15:38.203455 Marathon: State changed
2014/09/09 21:15:38.506437 Starting Goji on [::]:8000
2014/09/09 21:16:08.147463 Marathon: State changed
2014/09/09 21:16:08.449516 Starting Goji on [::]:8000
Looks like there is a problem but the error message isn't descriptive enough to diagnose it.
I built the pkg on Debian Wheezy and installed it successfully. However there is something wrong with the start script as my system tells me the files dont exist how do I get the service to start?
I'm not quite sure why or what the error is but I'm getting a panic in qzk.sinkSelfEvents:
goroutine 30 [chan receive]:
github.com/samuel/go-zookeeper/zk.(*Conn).request(0xc210068750, 0xc200000004, 0x641ea0, 0xc2100598e0, 0x641ee0, ...)
/tmp/src/github.com/samuel/go-zookeeper/zk/conn.go:595 +0x8f
github.com/samuel/go-zookeeper/zk.(*Conn).GetW(0xc210068750, 0xc210059e80, 0x1b, 0x0, 0x0, ...)
/tmp/src/github.com/samuel/go-zookeeper/zk/conn.go:638 +0x15f
github.com/QubitProducts/bamboo/qzk.func·004()
/tmp/src/github.com/QubitProducts/bamboo/qzk/qzk.go:82 +0x5d
created by github.com/QubitProducts/bamboo/qzk.sinkSelfEvents
/tmp/src/github.com/QubitProducts/bamboo/qzk/qzk.go:98 +0xfb
I've been trying with different Zookeeper settings and tried to purge all data in both /marathon
as well as /bamboo
(/bamboo/state
is set as DomainMapping path).
Do you have any pointers or can I provide anything else to help investigating?
Thanks!
Not sure if bug, but when running haproxy in the same container as bamboo, haproxy dies too and all requests all terminated.
It'd be cool if bamboo waited for zookeeper to be available doing nothing with haproxy config in the meantime.
Second solution would be running haproxy in separate container. If that's preferred - feel free to close the issue.
Some logs: https://gist.github.com/gregory90/00658376bc08dbe979da
I think current bamboo implementation does not take into account tasks service port(s).
Let me demonstrate it on one example with 2 exposed service ports:
Lets say I have following marathon app:
{
"id": "marattest-dev",
"cmd": "curl -q -o /app.py https://gist.githubusercontent.com/veverjak/b8607526f071f296bd8c/raw/3545a22d421d1f041ca5a8b305ea00f7edf967c6/get_file_content.py && python /app.py /etc/ip",
"instances": 3,
"cpus": 0.1,
"mem": 10,
"constraints": [],
"uris": ["/.dockercfg"],
"taskRateLimit": 1,
"container": {
"type": "DOCKER",
"volumes": [
{
"containerPath": "/var/lib/scripts/create_configs.sh",
"hostPath": "/var/lib/scripts/create_configs.sh",
"mode": "RO"
},
{
"containerPath": "/usr/bin/etcdctl",
"hostPath": "/usr/bin/etcdctl",
"mode": "RO"
},
{
"containerPath": "/etc/ip",
"hostPath": "/etc/ip",
"mode": "RO"
}
],
"docker": {
"image": "quay.io/appuri/get_file_content",
"portMappings": [
{ "containerPort": 5000, "hostPort": 0, "servicePort": 9001, "protocol": "tcp" },
{ "containerPort": 5001, "hostPort": 0, "servicePort": 9002, "protocol": "tcp" }
],
"network": "BRIDGE"
}
}
}
result for /v2/tasks gives:
{
"tasks": [
{
"appId": "/marattest-dev",
"id": "marattest-dev.46b9ab9c-6738-11e4-b85e-1645b8de8fb1",
"host": "ip-10-0-1-151.us-west-2.compute.internal",
"ports": [
31003,
31004
],
"startedAt": "2014-11-08T11:38:08.063Z",
"stagedAt": "2014-11-08T11:13:33.038Z",
"version": "2014-11-08T11:13:29.557Z",
"servicePorts": [
9001,
9002
]
},
{
"appId": "/marattest-dev",
"id": "marattest-dev.c2f5c97a-6736-11e4-b85e-1645b8de8fb1",
"host": "ip-10-0-1-149.us-west-2.compute.internal",
"ports": [
31006,
31007
],
"startedAt": "2014-11-08T11:38:08.051Z",
"stagedAt": "2014-11-08T11:02:42.474Z",
"version": "2014-11-08T11:02:39.555Z",
"servicePorts": [
9001,
9002
]
},
{
"appId": "/marattest-dev",
"id": "marattest-dev.4752421f-6738-11e4-b85e-1645b8de8fb1",
"host": "ip-10-0-1-149.us-west-2.compute.internal",
"ports": [
31008,
31009
],
"startedAt": "2014-11-08T11:38:08.022Z",
"stagedAt": "2014-11-08T11:13:34.037Z",
"version": "2014-11-08T11:13:29.557Z",
"servicePorts": [
9001,
9002
]
},
If I am using default marathon bash script to create haproxy.cfg, I get:
listen marattest-dev-9001
bind 0.0.0.0:9001
mode tcp
option tcplog
balance leastconn
server marattest-dev-3 ip-10-0-1-151.us-west-2.compute.internal:31003 check
server marattest-dev-2 ip-10-0-1-149.us-west-2.compute.internal:31006 check
server marattest-dev-1 ip-10-0-1-149.us-west-2.compute.internal:31008 check
listen marattest-dev-9002
bind 0.0.0.0:9002
mode tcp
option tcplog
balance leastconn
server marattest-dev-3 ip-10-0-1-151.us-west-2.compute.internal:31004 check
server marattest-dev-2 ip-10-0-1-149.us-west-2.compute.internal:31007 check
server marattest-dev-1 ip-10-0-1-149.us-west-2.compute.internal:31009 check
If I want the same from bamboo, it generates:
## listen ::marattest-dev_0
## bind *:0
## mode http
##
## balance leastconn
## option forwardfor
##
## server ::marattest-dev-ip-10-0-1-149.us-west-2.compute.internal-31006 ip-10-0-1-149.us-west-2.compute.internal:31006
## server ::marattest-dev-ip-10-0-1-151.us-west-2.compute.internal-31003 ip-10-0-1-151.us-west-2.compute.internal:31003
## server ::marattest-dev-ip-10-0-1-149.us-west-2.compute.internal-31008 ip-10-0-1-149.us-west-2.compute.internal:31008
#
It is because listen endpoint is created only per app and first app port is considered to be the only service port which I didn't define because I expected same behavior like default bash script has.
Workaround for me will be defining ports: []
per app and use only one port, which is fine for now, but I believe this can be easily fixed by adding proper structure in Create Apps.
The http handler of Bamboo does a lot of work, e.g. querying Zookeeper, reloading haproxy. This can take seconds on non-trivial installations.
Marathon sometimes sends a series of events to Marathon, each with its own POST request. This obviously times out (timeout 10 sec in Marathon). Hence, we are easily loosing events in Bamboo.
Our current setup involves running each app on a standard port on the haproxy layer
(this is how marathons old haproxy_cfg used to generate configs).
Apps are deployed with a specific port as a way to 'tag' them for a given service.
Would be useful to expose that port for Bamboo templates too
(I notice Bamboo currently only exposes 1 port for tasks so I propose to keep that behaviour).
Hi there.
When building bamboo.go, I got a build error on line 152(Filename: logPath)
According to these docs there is no "Filename".
Changing "Filename" to "Dir" fixed the error for me :)
Not sure if bug, but...
When running Bamboo on every mesos slave should I have callback in Marathon for each instance of Bamboo?
I thought one callback will be enough, but updates doesn't seem to propagate to other instances.
i am reading alot about mesos and marathon in specific since i'm interested in using mesos for managing long lived micro services. I would like to know what bamboo gives that mesos / marathon in it's latest version does not provide out of the box.
also - do you think that this project could be converged into marathon at some point or do you see this project continues to grow side by side.
finally - what is the current status of bamboo (known production installations etc) and what is your roadmap?
thanks for the very well and organised documentation!
Chen.
How does it handle the a/b deploys with marathon? Does it wait for all the new nodes to be spun up the updated or does it do bleeding deploys?
With portIndex
, Marathon health checks can be configured to run against any port associated with the task:
{
"path": "/api/health",
"portIndex": 0,
"protocol": "HTTP",
"gracePeriodSeconds": 30,
"intervalSeconds": 30,
"timeoutSeconds": 30,
"maxConsecutiveFailures": 3
}
The health check port isn't currently available within Bamboo - it's simply assumed that it's equal to $task.Port
. Consequently, configuring a Marathon app with a health check on a different port results in the app always failing HAProxy's checks.
Is there a way we could get something like $task.HealthCheckPort
?
server {{ $app.EscapedId}}...check inter 30000 check port {{ $task.HealthCheckPort }}
Hi,
sorry for posting this question, but I cannot get it to work properly:
I have a 3 master setup and the following config.json:
{
"Marathon": {
"Endpoint": "http://localhost:8080"
},
"Bamboo": {
"Endpoint": "http://masterN:8000", # this is the local IP of this master
"Zookeeper": {
"Host": "master0:2181,master1:2181,master2:2181",
"Path": "/marathon-haproxy/state",
"ReportingDelay": 5
}
},
"HAProxy": {
"TemplatePath": "/opt/bamboo/config/haproxy_template.cfg",
"OutputPath": "/etc/haproxy/haproxy.cfg",
"ReloadCommand": "PIDS=`pidof haproxy`; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done"
},
"StatsD": {
"Enabled": false,
"Host": "localhost:8125",
"Prefix": "bamboo-server.development."
}
}
explanation: as a bamboo endpoint I used each master's ip address, marathon endpoint is localhost and the zookeeper hosts are my three master ip addresses. Now my website switches between available and 503 in a millisecond manner. How do I make this setup work?
Just wondered if it's possible to look up the current marathon master via zookeeper somehow?
As it stands a single URL seems to be the only way to find marathon, which breaks
if that marathon node is down.
Is it possible to define MARATHON_ENDPOINT by specifying zookeeper path like you can when running for example mesos frameworks?
Path could be:
zk://10.0.1.45:2181,10.0.1.47:2181,10.0.1.46:2181/marathon
With the new 0.2.1 release, Mesos 0.2, Marathon 0.7.0rc3, I get the backtrace shown below just after startup. I have a Docker task in Marathon without ports. I registered that with the old bamboo version.
2014/09/19 06:48:47.715526 Using environment override MARATHON_ENDPOINT=http://10.0.0.1:5080
2014/09/19 06:48:47.718157 Using environment override BAMBOO_ENDPOINT=http://10.0.0.1:9000
2014/09/19 06:48:47.719109 Using environment override BAMBOO_ZK_HOST=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181
2014/09/19 06:48:47.719512 Using environment override BAMBOO_ZK_PATH=/bamboo
2014/09/19 06:48:47.853548 Starting Goji on [::]:8000
2014/09/19 06:48:47.865740 �[30;1m[f061086f5cad/MZc8sXsCMT-000001] �[0mStarted �[35;1mPOST �[0m�[34m"/api/marathon/event_callback" �[0mfrom 10.0.0.1:53546
2014/09/19 06:48:47.868047 subscribe_event => 2014-09-19T06:48:47.807Z
2014/09/19 06:48:48.039152 �[30;1m[f061086f5cad/MZc8sXsCMT-000001] �[0m�[31;1mpanic: runtime error: index out of range�[0m
/opt/go/src/github.com/zenazn/goji/web/middleware/recoverer.go:24 (0x55dbed)
com/zenazn/goji/web/middleware.func.006: debug.PrintStack()
/usr/lib/go/src/pkg/runtime/panic.c:248 (0x4149f6)
panic: runtime·newstackcall(d->fn, (byte*)d->args, d->siz);
/usr/lib/go/src/pkg/runtime/panic.c:482 (0x41529d)
panicstring: runtime·panic(err);
/usr/lib/go/src/pkg/runtime/panic.c:433 (0x4150b7)
panicindex: runtime·panicstring("index out of range");
/opt/go/src/github.com/QubitProducts/bamboo/services/marathon/marathon.go:151 (0x5e3f45)
com/QubitProducts/bamboo/services/marathon.createApps: simpleTasks = append(simpleTasks, Task{ Host: task.Host, Port: task.Ports[0] })
/opt/go/src/github.com/QubitProducts/bamboo/services/marathon/marathon.go:194 (0x5e406c)
com/QubitProducts/bamboo/services/marathon.FetchApps: apps := createApps(tasks, marathonApps)
/opt/go/src/github.com/QubitProducts/bamboo/services/haproxy/haproxy.go:19 (0x575f61)
com/QubitProducts/bamboo/services/haproxy.GetTemplateData: apps, _ := marathon.FetchApps(config.Marathon.Endpoint)
/opt/go/src/github.com/QubitProducts/bamboo/services/event_bus/event_handler.go:52 (0x48a3de)
com/QubitProducts/bamboo/services/event_bus.handleHAPUpdate: templateData := haproxy.GetTemplateData(conf, conn)
/opt/go/src/github.com/QubitProducts/bamboo/services/event_bus/event_handler.go:36 (0x48a0a8)
com/QubitProducts/bamboo/services/event_bus.(*Handlers).MarathonEventHandler: handleHAPUpdate(h.Conf, h.Zookeeper)
/opt/go/src/github.com/QubitProducts/bamboo/bamboo.go:52 (0x401d3d)
com/QubitProducts/bamboo/services/event_bus.*Handlers.MarathonEventHandler.fm: eventBus.Register(handlers.MarathonEventHandler)
/usr/lib/go/src/pkg/runtime/asm_amd64.s:339 (0x425832)
call32: CALLFN(call32, 32)
/usr/lib/go/src/pkg/reflect/value.go:474 (0x5428ab)
Value.call: call(fn, ptr, uint32(size))
/usr/lib/go/src/pkg/reflect/value.go:345 (0x54199d)
Value.Call: return v.call("Call", in)
/opt/go/src/github.com/QubitProducts/bamboo/services/event_bus/event_bus.go:66 (0x489d6a)
com/QubitProducts/bamboo/services/event_bus.(*EventBus).Publish: fn.Call(args[:])
/opt/go/src/github.com/QubitProducts/bamboo/api/event_subscription.go:29 (0x484e46)
com/QubitProducts/bamboo/api.(*EventSubscriptionAPI).Callback: sub.EventBus.Publish(event)
/opt/go/src/github.com/QubitProducts/bamboo/bamboo.go:77 (0x401f84)
com/QubitProducts/bamboo/api.*EventSubscriptionAPI.Callback.fm: goji.Post("/api/marathon/event_callback", eventSubAPI.Callback)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/router.go:113 (0x557c00)
com/zenazn/goji/web.netHTTPWrap.ServeHTTPC: h.Handler.ServeHTTP(w, r)
/opt/go/src/github.com/zenazn/goji/web/atomic.go:1 (0x55ad32)
com/zenazn/goji/web.(*netHTTPWrap).ServeHTTPC: // +build !appengine
/opt/go/src/github.com/zenazn/goji/web/router.go:210 (0x55849f)
com/zenazn/goji/web.routeMachine.route: rm.routes[si].handler.ServeHTTPC(*c, w, r)
/opt/go/src/github.com/zenazn/goji/web/router.go:251 (0x55880f)
com/zenazn/goji/web.(*router).route: methods, ok := rm.route(c, w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware.go:100 (0x5596d5)
com/zenazn/goji/web.func.002: router.route(&cs.C, w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/options.go:70 (0x55db3e)
com/zenazn/goji/web/middleware.func.004: h.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/recoverer.go:29 (0x55dd23)
com/zenazn/goji/web/middleware.func.007: h.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/logger.go:27 (0x55d8ec)
com/zenazn/goji/web/middleware.func.002: h.ServeHTTP(lw, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/request_id.go:68 (0x55dfdb)
com/zenazn/goji/web/middleware.func.008: h.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware.go:54 (0x554ee5)
com/zenazn/goji/web.(*cStack).ServeHTTP: s.m.ServeHTTP(w, r)
/opt/go/src/github.com/zenazn/goji/web/mux.go:77 (0x555fff)
com/zenazn/goji/web.(*Mux).ServeHTTP: stack.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1496 (0x4450f3)
(*ServeMux).ServeHTTP: h.ServeHTTP(w, r)
/opt/go/src/github.com/zenazn/goji/graceful/middleware.go:46 (0x561b99)
com/zenazn/goji/graceful.func.002: h.ServeHTTP(&fancyWriter{bw}, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
HandlerFunc.ServeHTTP: f(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1597 (0x4458fe)
serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/lib/go/src/pkg/net/http/server.go:1167 (0x4438a7)
(*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/lib/go/src/pkg/runtime/proc.c:1394 (0x418d00)
goexit: runtime·goexit(void)
2014/09/19 06:48:48.065102 �[30;1m[f061086f5cad/MZc8sXsCMT-000001] �[0mReturning �[31;1m500�[0m in �[32m199.234144ms�[0m
This is dependent on this issue in marathon: mesosphere/marathon#599
It would be great if I could configure Bamboo to automatically configure apps to hostnames via specifying metadata to Marathon.
For example, if I could specify a tag in Marathon of bamboo.hostname: my-app.example.com
and have bamboo automatically configure that application when it sees it.
This would allow us to automatically configure and provision the load balancing for services based off of metadata for the application itself, rather than making a separate api call to provision load balancing.
The output on an haproxy reload error is the following:
2014/10/20 13:02:31.794232 exit status 1
2014/10/20 13:02:31.794513 HAProxy: Configuration updated
The exit status has no context and the second line is plain wrong.
The web ui allows to define haproxy ACLs, but not the order they are applied, i.e. the order of use_backend lines in the config. I propose to add an order either by drag'n'drop in the UI or by a priority value in order to allow use cases like these:
Using a version of Bamboo that includes commit 6791f26, it comes with a change to the HAProxy configuration template backend section that treats a HTTP response with a non-empty body as a positive health signal. The configuration change in questions is this:
http-check expect rstring .*
What this means is that even if you specify an non-existing URI in the httpchk
option leading to a 404 response, you will still end up with a health check considered valid if the response contains some string like Not Found. In fact, this is the exact scenario that we ran into using Bamboo.
I couldn't fully understand from the git commit message what the reason was to make the change, so I am uncertain how a proper fix would look like. The http-check
regular expression could be extended to cover 2xx and 3xx responses (as in ^[23]\d{2}.*
), but that seems no different to leaving out the entire parameter (as httpchk
defaults to considering 2xx/3xx as valid).
Maybe @activars (the commit author) can shed some light on what the intention for the change was in the first place.
Thanks!
I have trouble adding app to bamboo from gui.
I've added app called /luigi-test and it says Missing app in Marathon.
I am running bamboo as docker container with command:
/usr/bin/docker run --rm --name bamboo -p 8000:8000 -p 80:80 -e MARATHON_ENDPOINT=10.0.1.149:8081 -e BAMBOO_ENDPOINT=10.0.1.151:8000 -e BAMBOO_ZK_HOST="10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181" -e BAMBOO_ZK_PATH=/bamboo quay.io/appuri/bamboo -bind=":8000" -config="config/production.example.json"
docker container logs:
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56 Starting update loop
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052755 Using environment override MARATHON_ENDPOINT=10.0.1.149:8081
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052823 Using environment override BAMBOO_ENDPOINT=10.0.1.151:8000
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052891 Using environment override BAMBOO_ZK_HOST=10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052949 Using environment override BAMBOO_ZK_PATH=/bamboo
When app added:
From bamboo logs:
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653968 Domain mapping: Stated changed
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653983 Queuing an haproxy update.
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.655489 HAProxy: Same content, no need to reload
From marathon, only relevant info I've found:
Nov 02 06:06:29 ip-10-0-1-149.us-west-2.compute.internal bash[6143]: [2014-11-02 06:06:29,066] INFO 10.0.1.151 - - [02/Nov/2014:06:06:29 +0000] "GET /v2/tasks HTTP/1.1" 200 252 "-" "curl/7.26.0" (mesosphere.chaos.http.ChaosRequestLog:15)
I can add marathon callback and call to /v2/tasks returns following:
curl -X GET -H "Content-Type: application/json" 10.0.1.149:8081/v2/tasks
{"tasks":[{"appId":"/luigi-test","id":"luigi-test.70583548-5f32-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31001],"startedAt":"2014-11-02T06:07:11.580Z","stagedAt":"2014-10-29T06:11:36.580Z","version":"2014-10-29T06:11:36.379Z","servicePorts":[9001]},{"appId":"/marathon-tcp-haproxy-dev","id":"marathon-tcp-haproxy-dev.9e92e270-606f-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31187],"startedAt":"2014-11-02T06:07:11.602Z","stagedAt":"2014-10-30T20:02:04.602Z","version":"2014-10-29T09:20:19.048Z","servicePorts":[30000]},{"appId":"/demo-dev","id":"demo-dev.f56bbeb8-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.563Z","stagedAt":"2014-10-28T21:25:37.088Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]},{"appId":"/demo-dev","id":"demo-dev.f69d39da-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.572Z","stagedAt":"2014-10-28T21:25:39.089Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]}]}%
I am using mesos 0.20.1
marathon 0.7.3
latest bamboo
Do you have any advice on how I could debug this further?
Can't build bamboo deb package
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.