go-graphite / go-carbon Goto Github PK

View Code? Open in Web Editor NEW

797.0 797.0 126.0 13.45 MB

Golang implementation of Graphite/Carbon server with classic architecture: Agent -> Cache -> Persister

License: MIT License

Makefile 0.86% Shell 0.89% Go 97.65% Python 0.45% Dockerfile 0.15%

carbon devops graphite hacktoberfest timeseries whisper

go-carbon's People

Contributors

Stargazers

Watchers

Forkers

godeep lees cbepx opentokix youtome cinderalla eyckwigo lucasjo gibheer szibis gitinsky henrypfhu halcyonardency fayizk1 dieterbe slach blurrcat johnclaus redbaron n3gster jriguera pasichnichenko drawks udhos 40a uberamd linbsd gophersgang bzed kokizzu onkarkadam7 iketutg qinguoan yunstanford ehlerst damienclaveau muhazzz manticorecao yanc0 hnakamur claudio-benfatto springerpe devinsmile azhiltsov cmoates brange korservick yanghongkjxy forcemain tameem92 bnikolaus bom-d-van civil joemiller maggnus guoyu07 gksinghjsr duckie gunnihinn rollno748 tigersean rpassmore dihaifeng kescott027 niteshjain gaoxp ellisvlad mantyr jjneely zombig olitvin friendlydan codelingobot aksakalli daanikus o awesomegolang mixton biox nyerup bookingcom avereha kingair zskarman sinopower saipraneethyss milekz grlvrl songjiao armandocerna deniszh aranc23 flucrezia oranenj enidvrenozaj zerosoul13 k3a loitho ashemez dvanders

go-carbon's Issues

Interface conversion error

Whenever I start the collectd deamon on any of my hosts, I get a number of errors in go-carbon's log, for example:

[2016-03-13 10:17:18] E [persister] UpdateMany /home/graphite/storage/inx/app12/interface/lo/if_octets/tx.wsp recovered: interface conversion: interface is runtime.errorString, not string
[2016-03-13 10:17:19] E [persister] UpdateMany /home/graphite/storage/inx/app12/processes/fork_rate.wsp recovered: interface conversion: interface is runtime.errorString, not string
[2016-03-13 10:17:28] E [persister] UpdateMany /home/graphite/storage/inx/app12/cpu/1/cpu/user.wsp recovered: interface conversion: interface is runtime.errorString, not string
[2016-03-13 10:17:38] E [persister] UpdateMany /home/graphite/storage/inx/app12/disk/xvda2/disk_time/write.wsp recovered: interface conversion: interface is runtime.errorString, not string

The common factor between these collectd plugins is that all are set as DERIVE datatypes. Note these errors only appear once while starting collectd.

Other GAUGE plugins, e.g. memory, do not lose any points in between collectd restarts while DERIVE based plugins, e.g. cpu, interface, have an empty gap because of this.

go-carbon does not stop gracefully and loss cache data

Я думаю процесс остановки должен происходить следующим образом:

Перестать принимать метрики. Закрыть соединения и освободить порты. Релей который стоит перед go-carbon, какое-то время сможет выполнять роль кэша.

Записать все данные из кэша, игнорируя настройку max-updates-per-second
Выйти.

Планируется такое реализовать?

Any plans to add relay support?

This project has literally saved us as we push towards 5M metrics per minute. Just curious if you plan to add relay support. We use haproxy today and we can split the data among multiple go-carbon instances but we'd like to send a full copy of the data to another location.

Thanks for releasing this.

Gary

Incorrect error handling in case of broken permissions to whisper file

Hello!

I have changed user for go-carbon from root to _graphite and fit interesting bug.

Actually files stored on disk with root permissions and go-carbon could not read it from user _graphite.

It's OK but it tried to create new files:

[2015-11-26 13:49:14] E [persister] Failed to create new whisper file /var/lib/graphite/whisper/carbon/agents/netflow_fv_ee/cache/checkpointTime.wsp: file already exists
[2015-11-26 13:49:14] E [persister] Failed to create new whisper file /var/lib/graphite/whisper/carbon/agents/netflow_fv_ee/cache/overflow.wsp: file already exists
[2015-11-26 13:49:14] E [persister] Failed to create new whisper file /var/lib/graphite/whisper/carbon/agents/netflow_fv_ee/cache/queries.wsp: file already exists
[2015-11-26 13:49:14] E [persister] Failed to create new whisper file /var/lib/graphite/whisper/carbon/agents/netflow_fv_ee/cache/metrics.wsp: file already exists
[2015-11-26 13:49:14] E [persister] Failed to create new whisper file /var/lib/graphite/whisper/carbon/agents/netflow_fv_ee/cache/size.wsp: file already exists

So I suggest to report this sort of error in more detailed way and do not try to create file in this case.

Thanks!

logrotate configuration

I have created a logrotate config file to be included in the deb and rpm packages.

The code is here:
jriguera@4030c0d

I think could be interesting to be included in the upstream repo via PR. Is it ok? or do you have other ideas of doing it?

Thanks!

Define a different endpoint for internal metrics

Perhaps I am the only one interested in this. I would like to be able to send the internal go-carbon metrics (carbon.agents.*) to a defined endpoint instead of only to the local persister. The use case would be sending to a load balancer/relay endpoint so that internal metrics get stored on multiple cache nodes.

How can I troubleshoot bad message further?

Hi all,

First of all thank you for all the hard work on this project.

I am trying to use go-carbon to directly replace carbon-cache.py. When I run it, I received bad message error. Example:

[2016-06-15 21:46:08] I bad message: "aaa.bbb.value [J@4442a5df 1466027166\n"

I changed the log level to debug, and see no new information.

Looks like this is the line where it breaks: https://github.com/lomik/go-carbon/blob/ece1e95d74d158a8b82c261ad616ee02480c59e8/points/points.go#L60

That [J@4442a5df is suspicious, does that mean I am receiving pickle format, instead of plain text?

Convert to use govendor for vendored libraries

This project predates govendor as a supported first party solution for vendoring in 3rd party libs and uses submodules instead. Now that golang has native vendor tooling/support it seems anachronistic to stay with the non-standard submodule setup.

NaN values being saved in Whisper datafiles

hi @lomik , With reference to https://bugs.launchpad.net/graphite/+bug/867882

Somehow I'm finding my whisper files to contain nan, and while this is valid from a python JSON spec, it's breaking the Grafana javascript code which parses the JSON from graphite. And causes an error and does not plot graphs.

Apparently carbon-cache was patched to eliminate such cases of writing nan to the whisper files. Can you look into this?

Agent metrics questions

Many metrics are reporting 0. Some are expected but some not. For example cache.queries, I wouldn't expect that to be 0. What is cache.checkpointTime and cache.metrics? Why are some of the *.active metrics 0 when they are enabled in the config?

Quick question re metrics in cache vs metrics on disk

Hi,

Does the storage schema affect the resolution of the metrics stored in the cache?

ie if the storage schema is '10s:100d' and I feed go-carbon metrics at a 1 second interval will a query to the cache result in metrics at 1 or 10 second resolution?

cheers.

FAIL: TestThrottleChan on some machines (i.e. in virtualbox)

When i try make deb package over

git checkout
git submodules init
git submodules update
make clean && make && make deb

i have following error on some machines

 --- FAIL: TestThrottleChan (8.00 seconds)
     assertions.go:156:
     Location:       throttle_test.go:38
     Error:          Should be true
     Messages:       perSecond: 1000, bw: 822

possible reason it happening when machine have too small CPU resources
do you have .deb PPA ? or maybe i can skip test? or maybe need increase minimum for throttling?

How compatible is the config file with regular carbon-cache.py?

Is this a drop-in replacement? This project looks great!

carbon.agents.{host}.persister.commitedPoints

Committed is spelled incorrectly in _commited_Points. ;-) If intentional please disregard.

rewrite-rules.conf support

Hello

Feature request.

It would be helpful if go-carbon supported the rewrite-rules from carbon-cache.

Roadmap

Hey there, love this project so far. Just wondering if you have a roadmap. I'd like to see the following features from carbon-cache implemented. Currently, the writes to my storage is higher than I'd like.

MAX_UPDATES_PER_SECOND
MAX_CREATES_PER_MINUTE
CARBON_METRIC_INTERVAL

Revert to github.com/sevlyar/go-daemon

It looks like this was originally forked and the fork set as a dependency while there was an open PR. However that PR has since been merged in sevlyar/go-daemon#6

support for multiple go-carbon caches

Hi,

I think this question is more suited for a mailing list, as its not actually an issue.

With graphite/carbon I am able to support multiple caches within a single process. Is their a way to do this with go-carbon or is the prescribed method to run multiple instances of go-carbon.

cheers,
max

go-carbon crashes when doing deep scan

I had go-carbon crash already once or twice in the last weeks, I didn't pay really attention and updated to the latest version (0.7-beta1) and it still occurs.

These are the "bad lines" found in the logs prior to the crash, as you can see this comes from regular security scans that run inside my company's network and are generated by something like nmap -sV --version-all -p XXXX mymachine

[2015-12-11 14:28:06] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:06] I bad message: "\x05\x04\x00\x01\x02\x80\x05\x01\x00\x03\n"
[2015-12-11 14:28:06] I bad message: "google.com\x00PGET / HTTP/1.0\r\n"
[2015-12-11 14:28:06] I bad message: "\r\n"
[2015-12-11 14:28:07] I [persister] doCheckpoint() {commitedPoints=173096, created=0, updateOperations=34786}
[2015-12-11 14:28:07] I [udp] doCheckpoint() {errors=0, incompleteReceived=0, metricsReceived=4643}
[2015-12-11 14:28:07] I [tcp] doCheckpoint() {active=2, errors=222, metricsReceived=15780}
[2015-12-11 14:28:07] I [pickle] doCheckpoint() {active=1, errors=11, metricsReceived=151529}
[2015-12-11 14:28:07] I [cache] doCheckpoint() {inputCapacity=51200, inputLenAfterCheckpoint=9, inputLenBeforeCheckpoint=3, metrics=4771, overflow=0, queries=1845, size=4887, time="1.936403ms"}
[2015-12-11 14:28:11] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:16] W [tcp] Unfinished line: []byte{0x4, 0x1, 0x0, 0x16, 0x7f, 0x0, 0x0, 0x1, 0x72, 0x6f, 0x6f, 0x74, 0x0}
[2015-12-11 14:28:16] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:21] W [tcp] Unfinished line: []byte{0x12, 0x1, 0x0, 0x34, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x15, 0x0, 0x6, 0x1, 0x0, 0x1b, 0x0, 0x1, 0x2, 0x0, 0x1c, 0x0, 0xc, 0x3, 0x0, 0x28, 0x0, 0x4, 0xff, 0x8, 0x0, 0x1, 0x55, 0x0, 0x0, 0x0, 0x4d, 0x53, 0x53, 0x51, 0x4c, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x0, 0x48, 0xf, 0x0, 0x0}
[2015-12-11 14:28:21] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:21] I bad message: "HELP\n"
[2015-12-11 14:28:21] I bad message: "\n"
[2015-12-11 14:28:21] I [pickle] Can't unpickle message: Pickle Machine failed on opcode:0x0. Stack size:0. Memo size:0. Cause:Opcode is invalid
[2015-12-11 14:28:21] I [pickle] Can't unpickle message: Pickle Machine failed on opcode:0x0. Stack size:0. Memo size:0. Cause:EOF
[2015-12-11 14:28:26] W [tcp] Unfinished line: []byte{0x0}
[2015-12-11 14:28:26] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:26] I bad message: "stats\r\n"
[2015-12-11 14:28:31] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:31] I bad message: "\x00\x00\x00\x01\x00\x00\x00\x13\x00\x00\x00\x02\x00\x00\x00$\x00\x00\x00\vservice_mgr\x00\x00\x00\x00\x02\x00\x00\x00\x13\x01\bscanner \x04\x05nmap \x06\x00\x00\x00\x00\x00\b\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x02\x00\x00\x00\n"
[2015-12-11 14:28:36] W [tcp] Unfinished line: []byte{0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x0, 0x3, 0x0, 0x0, 0x0, 0x4}
[2015-12-11 14:28:36] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:41] W [tcp] Unfinished line: []byte{0x0, 0x0, 0x0, 0x0, 0x44, 0x42, 0x32, 0x44, 0x41, 0x53, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x1, 0x4, 0x0, 0x0, 0x0, 0x10, 0x39, 0x7a, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0xc, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc, 0x0, 0x0, 0x0, 0xc, 0x0, 0x0, 0x0, 0x4}
[2015-12-11 14:28:41] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:46] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:46] W [tcp] Unfinished line: []byte{0x1, 0xc2, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0xb6, 0x1, 0x0, 0x0, 0x53, 0x51, 0x4c, 0x44, 0x42, 0x32, 0x52, 0x41, 0x0, 0x1, 0x0, 0x0, 0x4, 0x1, 0x1, 0x0, 0x5, 0x0, 0x1d, 0x0, 0x88, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x9, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x9, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x8, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x40, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x2, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x3, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x8, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x10, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x10, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x4, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x9, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x9, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x0, 0x3, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x4, 0x0, 0x0, 0x1, 0x0, 0x0, 0x80, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0x0, 0x40, 0x0, 0x0, 0x0, 0x0, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x0, 0xff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xe4, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7f}
[2015-12-11 14:28:51] W [pickle] Can't read message body: unexpected EOF
[2015-12-11 14:28:51] W [tcp] Unfinished line: []byte{0x43, 0x6c, 0x69, 0x65, 0x6e, 0x74, 0x20, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x20, 0x66, 0x6f, 0x72, 0x20, 0x50, 0x41, 0x52, 0x43, 0x20, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x20, 0x31, 0x20, 0x57, 0x69, 0x72, 0x65, 0x20, 0x45, 0x6e, 0x63, 0x72, 0x79, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x20, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x20, 0x31, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
[2015-12-11 14:28:51] I bad message: "<\x00K\x00\x00\x00 \x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\x00\x00\n"
[2015-12-11 14:28:51] I bad message: "\x04\xa0\xbeS\x03UR\x00\x00<\x00\x00\x00\x05\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1a\x00<\x00\x00\x00\x00\x00\n"

I was able to reproduce this once, but not again after that. I suspect there may be some kind of race there, as go-carbon is running with the following configuration options and receiving ~ 200k metrics.

[common]
...
max-cpu = 6

[whisper]
...
workers = 16

List of feature comparison to Graphite/Carbon

Hello,

This looks like an interesting project. Can you advise on any feature differences between the golang implementation and Graphite/Carbon please?

Thanks

Decimal numbers in log files instead of hexademical

Hello!

It's pretty hard to read hexademical numbers from log file:

[2015-11-26 14:45:31] I [tcp] doCheckpoint() {active=0, errors=0x0, metricsReceived=0x14af20}
[2015-11-26 14:47:31] I [tcp] doCheckpoint() {active=1, errors=0x0, metricsReceived=0x144b09}

Could you convert it to decimal representation for log file?

Thanks!

tests for points seem overly strict.

Currently there are a bunch of tests in points_test.go that appear to be designed for errors raised on minor problems with the input to ParseText. I notice however that it doesn't appear that ParseText implements the checks which the test expects.

I suppose that it makes sense to write the test first and then implement the expected behavior, however... in this case I think that the additional sanitization of input is unneeded. For instance "foo..bar" splits to fields like ["foo","","bar"] which, when building the path with filepath.join ends up collapsing out the empty path element. Similarly a leading "." ends up causing an empty path element between the storage directory and the whisper file name which similarly is collapsed.

The additional step of sanitization of these metric names would seem to be a wasted effort, even metric names like "foo.bar.." lead to only marginally confusing whisper file names like "foo/bar/.wsp" which arguably are valid and should be allowed.

Build rpm in centos 6 docker container

Hi,

I'm getting the following error when trying to build the rpm in a centos 6 docker container:

[root@3ff57d924fff go-carbon]# make rpm
mkdir -p tmp/
rm -rf tmp/go-carbon
mkdir -p tmp/go-carbon/
cp go-carbon tmp/go-carbon/go-carbon
./go-carbon --config-print-default > tmp/go-carbon/go-carbon.conf
cp deploy/go-carbon.init.centos tmp/go-carbon/go-carbon.init
cd tmp && tar czf go-carbon.tar.gz go-carbon/
cp deploy/buildrpm.sh tmp/buildrpm.sh
cd tmp && ./buildrpm.sh ../deploy/go-carbon.spec.centos `../go-carbon --version`
Executing(%prep): /bin/sh -e /root/go-carbon/tmp/rpm/tmp/rpm-tmp.Z2G3ku
+ umask 022
+ cd /root/go-carbon/tmp/rpm/BUILD
+ LANG=C
+ export LANG
+ unset DISPLAY
+ rm -rf /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64
+ cd /root/go-carbon/tmp/rpm/BUILD
+ rm -rf go-carbon
+ /bin/tar -xvvf -
+ /usr/bin/gzip -dc /root/go-carbon/tmp/go-carbon.tar.gz
drwxr-xr-x root/root         0 2015-10-14 14:12 go-carbon/
-rw-r--r-- root/root       641 2015-10-14 14:12 go-carbon/go-carbon.conf
-rwxr-xr-x root/root      1836 2015-10-14 14:12 go-carbon/go-carbon.init
-rwxr-xr-x root/root  10299680 2015-10-14 14:12 go-carbon/go-carbon
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd go-carbon
+ /bin/chmod -Rf a+rX,u+w,g-w,o-w .
+ exit 0
Executing(%build): /bin/sh -e /root/go-carbon/tmp/rpm/tmp/rpm-tmp.H4Qq6I
+ umask 022
+ cd /root/go-carbon/tmp/rpm/BUILD
+ cd go-carbon
+ LANG=C
+ export LANG
+ unset DISPLAY
+ exit 0
Executing(%install): /bin/sh -e /root/go-carbon/tmp/rpm/tmp/rpm-tmp.U4cRSX
+ umask 022
+ cd /root/go-carbon/tmp/rpm/BUILD
+ '[' /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64 '!=' / ']'
+ rm -rf /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64
++ dirname /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64
+ mkdir -p /root/go-carbon/tmp/rpm/BUILDROOT
+ mkdir /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64
+ cd go-carbon
+ LANG=C
+ export LANG
+ unset DISPLAY
+ '[' /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64 '!=' / ']'
+ rm -fr /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64
+ /bin/mkdir -p /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64/usr/local/bin
+ /usr/bin/install -pD -m 755 go-carbon /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64//usr/local/bin/go-carbon
Name:       go-carbon
+ /usr/bin/install -pD -m 644 go-carbon.conf /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64//usr/local/etc/go-carbon.conf
+ /usr/bin/install -pD -m 755 go-carbon.init /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64/etc/init.d/go-carbon
+ /usr/lib/rpm/find-debuginfo.sh --strict-build-id /root/go-carbon/tmp/rpm/BUILD/go-carbon
extracting debug info from /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64/usr/local/bin/go-carbon
/usr/lib/rpm/debugedit: /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64/usr/local/bin/go-carbon: Unknown debugging section .debug_gdb_scripts
*** ERROR: No build ID note found in /root/go-carbon/tmp/rpm/BUILDROOT/go-carbon-0.5.4-1.el6.x86_64/usr/local/bin/go-carbon
error: Bad exit status from /root/go-carbon/tmp/rpm/tmp/rpm-tmp.U4cRSX (%install)


RPM build errors:
    Bad exit status from /root/go-carbon/tmp/rpm/tmp/rpm-tmp.U4cRSX (%install)
make: *** [rpm] Error 1

After adding %define debug_package %{nil} to the spec file I was able to build the rpm.

Also by just reading the docs was not obvious that you provide an RPM in the releases page. Adding that to the docs would be useful!.

Thanks
Nick

CommittedPoints equivalent to native graphite CommittedPoints?

Can someone clarify/confirm if the go-carbon CommittedPoints internal persister counter is equivalent to the py-carbon counter of the same name?
I'm trying to get go-carbon in place to replace py-carbon on our internal (6m points min) setup. On identical hardware, for py-carbon I get 375k committedpoints a minute per server (running 4 instances). On go-carbon the committed points averages <200k per minute.

Server is using EBS SSD, 16GB RAM, 4 proc

I've tried the following combinations of settings:

max-updates-per-second = [tried 0, 500, 1000, 10000]
max-cpu = [3, 4] note: 4 cpu server
workers = [6, 8, 18]
max-size = [1000000, 5000000]
write-strategy = [max, sorted]
input-buffer = [51200, 512000, 5120000]
max-message-size = 67108864

The server is only getting metrics from a carbon-relay-ng server, and all stats are over pickle since this is running next to the py-carbon servers. Right now metrics are being dumped in at almost 800k metrics a minute (caches full, overflow shows ~600k metrics a min dropped). Are there other settings I should be aware of that would cause a significant performance differences, or are the metrics representing different things so I'm not actually comparing apples to apples?

go-carbon built yesterday so recent code release.

separate `bad mesage` from `doCheckpoint()` log messages

Right now bad message is reported with info level same as doCheckpoint, it would be really nice it they were separated,so that only doCheckpoint() messages were logged.

One option would be to introduce notice severity level betwern info and warning and make doCheckpoint() to log using notice. that way, by setting log-level=notice it would be possible to shut bad message messages.

Another option would be to move bad message to debug level

Enabling reload section on init script

Hi,

As currently the program supports configuration reloading (at least for persister config), can I submit a PR to enable the feature in the init script?

wrong carbonlink request from

Я всё время получаю эту ошибку, и графики периодически отстают.

[2015-06-09 20:07:01] W [carbonlink] wrong carbonlink request from: 127.0.0.1:41268
[2015-06-09 20:07:01] W [carbonlink] wrong carbonlink request from: 127.0.0.1:41276
[2015-06-09 20:07:01] W [carbonlink] wrong carbonlink request from: 127.0.0.1:41269

Как отключить такие сообщения в логе? У меня 9 Гигабайт логов за сутки.

[2015-06-09 20:13:16] W bad message: "mongodb.gra.databases.gra.users.wiredTiger.btree.row-store internal pages 0 1433862609\n"

Я так понимаю что пока reload не поддерживается? Делаю kill -HUP $(pgrep go-carbon), в логе вижу сообщение о принятии, но ничего не меняется.

Aggregation not worked?

i have following go-carbon.conf

[common]
user = "monitor"
logfile = "/var/log/go-carbon/go-carbon.log"
log-level = "info"
graph-prefix = "carbon.agents.{host}."
max-cpu = 8
metric-interval = "1m0s"

[whisper]
data-dir = "/opt/graphite/storage/whisper/"
schemas-file = "/opt/graphite/conf/storage-schemas.conf"
aggregation-file = "/opt/graphite/conf/storage-aggregation.conf"
workers = 8
max-updates-per-second = 0
sparse-create = true
enabled = true

[cache]
max-size = 1000000
input-buffer = 1000000

[udp]
listen = "127.0.0.1:2013"
enabled = true
log-incomplete = false

[tcp]
listen = "127.0.0.1:2013"
enabled = true
log-incomplete = true

[pickle]
listen = "127.0.0.1:2014"
max-message-size = 67108864
enabled = true

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
read-timeout = "30s"
query-timeout = "100ms"

[pprof]
listen = "localhost:7007"
enabled = false

and following aggregation rules

pinba.total.traffic_per_sec (60) = sum pinba.<host>.<server>.<script>.traffic_per_sec
pinba.total.req_count (60) = sum pinba.<host>.<server>.<script>.req_count

pinba.total.req_50_percentile (60) = max pinba.*.*.*.req_50_percentile
pinba.total.req_90_percentile (60) = max pinba.*.*.*.req_90_percentile
pinba.total.req_95_percentile (60) = max pinba.*.*.*.req_95_percentile
pinba.total.req_98_percentile (60) = max pinba.*.*.*.req_98_percentile

pinba.total.error_count   (60) = sum pinba.*.*.*.error_count
pinba.total.error_percent (60) = avg pinba.*.*.*.error_percent

pinba.total.tag_by_req (60) = sum pinba.*.*.*.*.*.tag_by_req_avg
pinba.total.tag_req_percent (60) = avg pinba.*.*.*.*.*.tag_req_percent

pinba.total.req_less_02s (60) = sum pinba.*.*.*.req_less_02s
pinba.total.req_02s_05s (60) = sum pinba.*.*.*.req_02s_05s
pinba.total.req_05s_1s (60) = sum pinba.*.*.*.req_05s_1s
pinba.total.req_1s_2s (60) = sum pinba.*.*.*.req_1s_2s
pinba.total.req_more_2s (60) = sum pinba.*.*.*.req_more_2s

pinba.total.tag_less_01s (60) = sum pinba.*.*.*.*.*.tag_less_01s
pinba.total.tag_01s_02s (60) = sum pinba.*.*.*.*.*.tag_01s_02s
pinba.total.tag_02s_05s (60) = sum pinba.*.*.*.*.*.tag_02s_05s
pinba.total.tag_05s_1s (60) = sum pinba.*.*.*.*.*.tag_05s_1s
pinba.total.tag_more_1s (60) = sum pinba.*.*.*.*.*.tag_more_1s

pinba.total.tag_50_percentile (60) = max pinba.*.*.*.*.*.tag_50_percentile
pinba.total.tag_90_percentile (60) = max pinba.*.*.*.*.*.tag_90_percentile
pinba.total.tag_95_percentile (60) = max pinba.*.*.*.*.*.tag_95_percentile
pinba.total.tag_98_percentile (60) = max pinba.*.*.*.*.*.tag_98_percentile

pinba.* metrics received on port 2013 and i see this traffic over

tcpdump -i lo -w go-carbon.pcap port 2013 and host 127.0.0.1

see http://take.ms/qS34T

but pinba.total.* metrics in wsp files don't updates

in go-carbon.log i don't see any errors about incorrect aggregation rules, how i can debug and fix it this problem?

Add runtime tunables to internal metrics

It would be nice if various of the runtime tunables were exported as internal metrics. For instance cache max-size and input-buffer would more easily allow for you to create dashboard which show that you are approaching an internal limit before you start dropping metrics. Or whisper workers or max -cpu would allow you to better understand how work is divided and how it scales when tuning. These become especially informative if your carbon/graphite stack is evolving over time as it allows for easier growth trend analysis.

Whisper updates not correct with many points in cache

This is pretty much a continuation to this issue: grobian/carbon-c-relay#123
Currently we keep a large cache around because the writes are getting cached by the OS and causing all services on the box to grind to a halt. We currently do 19 points per update and 3000 updates per minute.

I have a whisper file setup with two archives:
10s:1m
1m:2y

These are configured to aggregate by sum. When end result data points should be the same they are not. When only one archive is used there does not seem to be an issue.

creates.log

I am planning on migrating a carbon-cache (~80k points/sec) setup to go-carbon.

I like to track what whisper files are being created to detect if high cardinality identifiers (e.g. session-ids, user-ids, order-id) are used in the metrics names.

Example of carbon-cache's creates.log:

08/06/2016 08:56:49 :: creating database file /opt/graphite/storage/whisper/a/b/c.wsp (archive=[(60, 11520), (300, 9216), (900, 17280), (1800, 27600)] xff=None agg=None)

Could you maybe implement something similar in go-carbon?

Does go-carbon have any notion of stalling?

We have this situation where relay's metricsQueued climbs to almost a million. And it looks like go-carbon is not even pressured at all.

Does go-carbon have a meter that tells carbon-c-relay to back off after a certain size?

We think that our input-buffer is too conservative (51200), but looking at the code, it looks like input-buffer is used to create channel capacity, so we are not sure if increasing it to 1 million would help. Are we on the right path?

Cut a new version?

It looks like we are about 15 commits ahead of the last version and includes:

new feature
- sorted write strategy
distro packaging changes
- reload in redhat init
- debian init script arg passing fixed
vendored library changes
- revert to github.com/sevylar/go-daemon
- include github.com/kardianos/osext

Seems, to me, like we should tag 0.8.0

Is there any roadmap or outstanding issues to resolve before then?

Production ready?

Go-carbon looks very clean and promising.

Before replacing an array of 8 carbon-cache daemons I was wondering if you believe go-carbon is ready. Do you (or anyone else) use go-carbon in a production setup? How many datapoints/sec does it handle? My $dayjob graphite cluster currently handles 45k points/sec per node, for some 2.6M metrics in 1.6T on flash.

Thank you so much.

Sparse .wsp creation

Hi there,

Have recently started to use go-cabon on our secondary metric store for benchmark purposes (am quite excited at the prospect of not having to depend on the python caches) - am getting some very promising results so far.

A minor point, based on our tiered storage layout, we get quite tight for space without sparse whisper file creation enabled - is this option something that is on the roadmap for go-cabon?

Tuning Parameters and metrics questions

We're testing go-carbon as a possible replacement for python carbon. Couple of questions have come up:

Do we need to ensure that persister.committedPoints is the same as pickle.metricsReceived (we're feeding go-carbon from a relay...). I notice that the cache size immediately goes up to max if we keep IO down by setting max-updates-per-second. If committedPoints is less than metricsReceived, are we losing data?
Is it possible to run multiple go-carbon instances on a single server? Is this even desirable?

Finally, any guidance you can give on tuning - we have set max-updates-per-second, max-cpu and cache max-size to various values during testing, but it's not completely clear what the "best" way to set these is.

Thanks!

Support aggregation-rules.conf

First of all, thanks for sharing a nice software!

I would like to sum up multiple metrics (the same metric from multiple servers) and save to another metric. I would like to keep the original metrics from multiple servers and the sum alongside them.
I read the document of the original carbon and found it is supported in aggregation-rules.conf.

So I wish a support of aggregation-rules.conf in go-carbon too!
If you are busy, I will try to write a pull request for that.

Clustering issue

Trying to scale out my cluster, I saw that clusterization seems to fail.
Any idea or best practices ? Fyi, a similar setup works fine with python-carbon

Here is the cache config

[common]
# Run as user. Works only in daemon mode
user = ""
# If logfile is empty use stderr
logfile = "/var/log/go-carbon/go-carbon.log"
# Logging error level. Valid values: "debug", "info", "warn", "warning", "error"
log-level = "info"
# Prefix for store all internal go-carbon graphs. Supported macroses: {host}
graph-prefix = "carbon.agents.{host}."
# Interval of storing internal metrics. Like CARBON_METRIC_INTERVAL
metric-interval = "1m0s"
# Increase for configuration with multi persisters
max-cpu = 1

[whisper]
data-dir = "/var/lib/graphite/storage/whisper/"
# http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-schemas-conf. Required
schemas-file = "/data/graphite/schemas"
# http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-aggregation-conf. Optional
aggregation-file = ""
# Workers count. Metrics sharded by "crc32(metricName) % workers"
workers = 8
# Limits the number of whisper update_many() calls per second. 0 - no limit
max-updates-per-second = 0
# Sparse file creation
sparse-create = false
enabled = true

[cache]
# Limit of in-memory stored points (not metrics)
max-size = 1000000
# Capacity of queue between receivers and cache
input-buffer = 51200

[udp]
listen = ":2003"
enabled = true
# Enable optional logging of incomplete messages (chunked by MTU)
log-incomplete = false

[tcp]
listen = ":2003"
enabled = false

[pickle]
listen = ":2004"
enabled = true
# Limit message size for prevent memory overflow
max-message-size = 67108864

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
# Close inactive connections after "read-timeout"
read-timeout = "30s"
# Return empty result if cache not reply
query-timeout = "100ms"

[pprof]
listen = "localhost:7007"
enabled = false

And the local_settings :

## Graphite local_settings.py
# Edit this file to customize the default Graphite webapp settings
#
# Additional customizations to Django settings can be added to this file as well

#####################################
# General Configuration #
#####################################
# Set this to a long, random unique string to use as a secret key for this
# install. This key is used for salting of hashes used in auth tokens,
# CRSF middleware, cookie storage, etc. This should be set identically among
# instances if used behind a load balancer.
SECRET_KEY = '4LsegWYnFKeGt69YwULahIJeVtlBic7pKIQhMPQ293zE'

# In Django 1.5+ set this to the list of hosts your graphite instances is
# accessible as. See:
# https://docs.djangoproject.com/en/dev/ref/settings/#std:setting-ALLOWED_HOSTS
#ALLOWED_HOSTS = [ '*' ]

# Set your local timezone (Django's default is America/Chicago)
# If your graphs appear to be offset by a couple hours then this probably
# needs to be explicitly set to your local timezone.
TIME_ZONE = 'Europe/Paris'

# Override this to provide documentation specific to your Graphite deployment
#DOCUMENTATION_URL = "http://graphite.readthedocs.org/"

# Logging
LOG_RENDERING_PERFORMANCE = True
LOG_CACHE_PERFORMANCE = True
# LOG_METRIC_ACCESS = True

# Enable full debug page display on exceptions (Internal Server Error pages)
DEBUG = True

# If using RRD files and rrdcached, set to the address or socket of the daemon
#FLUSHRRDCACHED = 'unix:/var/run/rrdcached.sock'

# This lists the memcached servers that will be used by this webapp.
# If you have a cluster of webapps you should ensure all of them
# have the *exact* same value for this setting. That will maximize cache
# efficiency. Setting MEMCACHE_HOSTS to be empty will turn off use of
# memcached entirely.
#
# You should not use the loopback address (127.0.0.1) here if using clustering
# as every webapp in the cluster should use the exact same values to prevent
# unneeded cache misses. Set to [] to disable caching of images and fetched data
MEMCACHE_HOSTS = ['10.66.66.66:11211']
DEFAULT_CACHE_DURATION = 6000 # Cache images and data for 1 minute


#####################################
# Filesystem Paths #
#####################################
# Change only GRAPHITE_ROOT if your install is merely shifted from /opt/graphite
# to somewhere else
#GRAPHITE_ROOT = '/opt/graphite'

# Most installs done outside of a separate tree such as /opt/graphite will only
# need to change these three settings. Note that the default settings for each
# of these is relative to GRAPHITE_ROOT
#CONF_DIR = '/opt/graphite/conf'
#STORAGE_DIR = '/opt/graphite/storage'
#CONTENT_DIR = '/opt/graphite/webapp/content'

# To further or fully customize the paths, modify the following. Note that the
# default settings for each of these are relative to CONF_DIR and STORAGE_DIR
#
## Webapp config files
#DASHBOARD_CONF = '/opt/graphite/conf/dashboard.conf'
#GRAPHTEMPLATES_CONF = '/opt/graphite/conf/graphTemplates.conf'

## Data directories
# NOTE: If any directory is unreadable in DATA_DIRS it will break metric browsing
WHISPER_DIR = '/var/lib/graphite/storage/whisper/'
#RRD_DIR = '/opt/graphite/storage/rrd'
#DATA_DIRS = [WHISPER_DIR, RRD_DIR] # Default: set from the above variables
LOG_DIR = '/var/log/'
#INDEX_FILE = '/opt/graphite/storage/index'  # Search index file


#####################################
# Email Configuration #
#####################################
# This is used for emailing rendered Graphs
# Default backend is SMTP
#EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'
#EMAIL_HOST = 'localhost'
#EMAIL_PORT = 25
#EMAIL_HOST_USER = ''
#EMAIL_HOST_PASSWORD = ''
#EMAIL_USE_TLS = False
# To drop emails on the floor, enable the Dummy backend:
#EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend'


#####################################
# Authentication Configuration #
#####################################
## LDAP / ActiveDirectory authentication setup
#USE_LDAP_AUTH = True
#LDAP_SERVER = "ldap.mycompany.com"
#LDAP_PORT = 389
#   OR
#LDAP_URI = "ldaps://ldap.mycompany.com:636"
#LDAP_SEARCH_BASE = "OU=users,DC=mycompany,DC=com"
#LDAP_BASE_USER = "CN=some_readonly_account,DC=mycompany,DC=com"
#LDAP_BASE_PASS = "readonly_account_password"
#LDAP_USER_QUERY = "(username=%s)"  #For Active Directory use "(sAMAccountName=%s)"
#
# If you want to further customize the ldap connection options you should
# directly use ldap.set_option to set the ldap module's global options.
# For example:
#
#import ldap
#ldap.set_option(ldap.OPT_X_TLS_REQUIRE_CERT, ldap.OPT_X_TLS_ALLOW)
#ldap.set_option(ldap.OPT_X_TLS_CACERTDIR, "/etc/ssl/ca")
#ldap.set_option(ldap.OPT_X_TLS_CERTFILE, "/etc/ssl/mycert.pem")
#ldap.set_option(ldap.OPT_X_TLS_KEYFILE, "/etc/ssl/mykey.pem")
# See http://www.python-ldap.org/ for further details on these options.

## REMOTE_USER authentication. See: https://docs.djangoproject.com/en/dev/howto/auth-remote-user/
#USE_REMOTE_USER_AUTHENTICATION = True

# Override the URL for the login link (e.g. for django_openid_auth)
#LOGIN_URL = '/account/login'


##########################
# Database Configuration #
##########################
# By default sqlite is used. If you cluster multiple webapps you will need
# to setup an external database (such as MySQL) and configure all of the webapp
# instances to use the same database. Note that this database is only used to store
# Django models such as saved graphs, dashboards, user preferences, etc.
# Metric data is not stored here.
#
# DO NOT FORGET TO RUN 'manage.py syncdb' AFTER SETTING UP A NEW DATABASE
#
# The following built-in database engines are available:
#  django.db.backends.postgresql          # Removed in Django 1.4
#  django.db.backends.postgresql_psycopg2
#  django.db.backends.mysql
#  django.db.backends.sqlite3
#  django.db.backends.oracle
#
# The default is 'django.db.backends.sqlite3' with file 'graphite.db'
# located in STORAGE_DIR
#
#DATABASES = {
#    'default': {
#        'NAME': '/opt/graphite/storage/graphite.db',
#        'ENGINE': 'django.db.backends.sqlite3',
#        'USER': '',
#        'PASSWORD': '',
#        'HOST': '',
#        'PORT': ''
#    }
#}
#


#########################
# Cluster Configuration #
#########################
# (To avoid excessive DNS lookups you want to stick to using IP addresses only in this entire section)
#
# This should list the IP address (and optionally port) of the webapp on each
# remote server in the cluster. These servers must each have local access to
# metric data. Note that the first server to return a match for a query will be
# used.
CLUSTER_SERVERS = ["10.66.66.66:65533", "10.66.66.74:80", "10.66.66.66:65534"]

## These are timeout values (in seconds) for requests to remote webapps
REMOTE_STORE_FETCH_TIMEOUT = 5   # Timeout to fetch series data
REMOTE_STORE_FIND_TIMEOUT =  10 # Timeout for metric find requests
REMOTE_STORE_RETRY_DELAY = 2   # Time before retrying a failed remote webapp
REMOTE_FIND_CACHE_DURATION = 30000 # Time to cache remote metric find results

## Prefetch cache
# set to True to fetch all metrics using a single http request per remote server
# instead of one http request per target, per remote server.
# Especially useful when generating graphs with more than 4-5 targets or if
# there's significant latency between this server and the backends. (>20ms)
REMOTE_PREFETCH_DATA = False


## Remote rendering settings
# Set to True to enable rendering of Graphs on a remote webapp
REMOTE_RENDERING = False
# List of IP (and optionally port) of the webapp on each remote server that
# will be used for rendering. Note that each rendering host should have local
# access to metric data or should have CLUSTER_SERVERS configured
RENDERING_HOSTS = ["10.66.66.66:65534"]
#RENDERING_HOSTS = ["10.66.66.66:65534"]
REMOTE_RENDER_CONNECT_TIMEOUT = 120.0

# If you are running multiple carbon-caches on this machine (typically behind a relay using
# consistent hashing), you'll need to list the ip address, cache query port, and instance name of each carbon-cache
# instance on the local machine (NOT every carbon-cache in the entire cluster). The default cache query port is 7002
# and a common scheme is to use 7102 for instance b, 7202 for instance c, etc.
#
# You *should* use 127.0.0.1 here in most cases
#CARBONLINK_HOSTS = ["127.0.0.1:7002:a", "127.0.0.1:7102:b", "127.0.0.1:7202:c"]
CARBONLINK_HOSTS = ["127.0.0.1:7002"]
CARBONLINK_TIMEOUT = 1.0
# Using 'query-bulk' queries for carbon
# It's more effective, but python-carbon 0.9.13 (or latest from 0.9.x branch) is required
# See https://github.com/graphite-project/carbon/pull/132 for details
CARBONLINK_QUERY_BULK = True

#####################################
# Additional Django Settings #
#####################################
# Uncomment the following line for direct access to Django settings such as
# MIDDLEWARE_CLASSES or APPS
#from graphite.app_settings import *

any ideas?

Strange null values when pulling data with Graphite Django app

Hello!

I'm pulling data from go-carbon with Graphite's standard django app and getting strange results:

curl 'http://xxxxxxx/render?from=-10minutes&format=json&target=host.dst'
[{"target": "akamai.dst", "datapoints": [[null, 1448309460], [null, 1448309520], [null, 1448309580], [null, 1448309640], [null, 1448309700], [null, 1448309760], [null, 1448309820], [null, 1448309880], [null, 1448309940], [null, 1448310000]]}]

Example of put data request:

sudo tcpdump -i lo 'port 2003' -A|grep host.dst
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
host.dst 43 1448310192
host.dst 40 1448310193

For some strange reasons I'm getting null instead my non zero 64 bit integer data. Do you have some ideas how to isolate problem?

Graphite-Web compatibility map?

Hi @lomik I'm really surprised with this project , I would like to test it , but I need to know first if has been tested with both current graphite-web branches.

master (0.10.X) ?
0.9.x ?

I need also test with my currently working whisper files, can go-carbon work with already created files with original whisper python tools?

Datapoints (for new metric) not flushed to disk, but kept in cache memory

Curious case.

Our setup:

2 nodes
node1 runs go-carbon
node2 runs 12 carbon-cache daemons
carbon-c-relay on both nodes sending each line received to go-carbon on node1 and one of the carbons on node2

Expected: so each node should have the same whisperfiles

In reality: for 99.99% of all metrics its true, but we've found an edge case.

What happens:

User sends a new metric with sporadic datapoints
Points have been flushed to disk by carbon-cache and whisper fils have been created
Go-carbon hasn't flushed them yet, so no whisper files exist

It seems go-carbon is holding on to the datapoints in cache memory.
Carbonlink query on go-carbon returns this:

[(1467718603, 0.0),
(1467720962, 5.0),
(1467722489, 5.0),
(1467724363, 5.0),
(1467726788, 5.0),
(1467727826, 5.0),
(1467790086, 2106.0)]

That 1st timestamp is 22 hours ago.

Using this script to query:

#!/usr/bin/env python
import sys
sys.path.append('/opt/graphite/webapp')

import os
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "graphite.settings")
from graphite import settings
settings.LOG_DIR = '.'
del settings.ADMIN_MEDIA_PREFIX
del settings.CACHE_BACKEND

from graphite.render import datalib as dl

def query(metric):
  hosts = []
  for e in settings.CARBONLINK_HOSTS:
    host, port, instance = e.split(':')
    hosts.append((host, int(port), instance))

  p = dl.CarbonLinkPool(hosts, 1.0)
  print p.query(metric)

for m in sys.argv[1:]:
  query(m)

Frequent carbonlink errors

I am seeing a number of errors similar to the following:

[2015-11-19 13:56:51] D [carbonlink] read carbonlink request from 127.0.0.1:32800: Can't read message length: read tcp 127.0.0.1:7002->127.0.0.1:32800: i/o timeout
[2015-11-19 13:56:51] D [carbonlink] read carbonlink request from 127.0.0.1:32998: Can't read message length: read tcp 127.0.0.1:7002->127.0.0.1:32998: i/o timeout

I assume this is my fault. Can you guide me any on how to fix this issue?

go vet reports problems

/lomik/go-carbon (master=)$ go vet ./...
cache/cache_test.go:46: unreachable code
exit status 1
logging/logger.go:34: unrecognized printf flag for verb 's': '#'
logging/logger.go:105: unrecognized printf flag for verb 's': '#'
logging/rotate_test.go:18: unreachable code
exit status 1
persister/whisper_test.go:116: unrecognized printf verb 'a'
exit status 1
receiver/udp.go:262: unreachable code
exit status 1

crash of 0.7.2 (and 0.7.1)

Just had go-carbon-0.7.1-1.el6.x86_64 crashing.

I have included all info I could think. If you need more information or want me to run some additional diagnostics, please let me know.

Crash occurred at 22:01. The logs is totally silent:

[2016-06-23 21:58:24] I [carbonlink] Cache no reply (100ms timeout)
[2016-06-23 22:25:25] I started

I've been running go-carbon for some days ramping up the volume. This afternoon I put the peddle to the floor. Unsure if this is related, but its the biggest recent change.

Some numbers:

tcp.metricsReceived = 5.1M (~ 80k/sec)
cache.size = 21M
cache.metrics = 3M (so ~7 datapoints/metric in cache)
persister.updateOperations = 320k (~5300 update/sec == 90M/s write on flash)

My config:

[common]
user = ""
logfile = "/srv/graphite/log/go-carbon/go-carbon.log"
log-level = "info"
graph-prefix = "carbon.agents.{host}."
max-cpu = 4
metric-interval = "1m0s"

[whisper]
data-dir = "/srv/graphite/whisper/"
schemas-file = "/opt/graphite/conf/storage-schemas.conf"
aggregation-file = ""
workers = 4
max-updates-per-second = 7500
enabled = true

[cache]
max-size = 0
input-buffer = 5120000

[udp]
listen = ":2003"
enabled = false
log-incomplete = false

[tcp]
listen = "a.b.c.d:12003"
enabled = true

[pickle]
listen = ":2004"
enabled = false
max-message-size = 67108864

[carbonlink]
listen = "a.b.c.d:7002"
enabled = true
read-timeout = "30s"
query-timeout = "100ms"

[pprof]
listen = "localhost:7007"
enabled = false

Top output for the go-carbon pid:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
14950 root      20   0 3671m 2.9g 4176 R 53.6  2.3  17:57.26 go-carbon
14948 root      20   0 3671m 2.9g 4176 S 40.3  2.3  18:29.81 go-carbon
14951 root      20   0 3671m 2.9g 4176 S 31.3  2.3  20:42.91 go-carbon
14947 root      20   0 3671m 2.9g 4176 R 29.0  2.3  20:48.62 go-carbon
14976 root      20   0 3671m 2.9g 4176 S 25.0  2.3  20:28.41 go-carbon
14953 root      20   0 3671m 2.9g 4176 S 24.0  2.3  20:45.73 go-carbon
14957 root      20   0 3671m 2.9g 4176 R 22.3  2.3  19:41.77 go-carbon
14941 root      20   0 3671m 2.9g 4176 R 13.3  2.3  19:32.04 go-carbon
14943 root      20   0 3671m 2.9g 4176 S  0.0  2.3   1:10.92 go-carbon
14944 root      20   0 3671m 2.9g 4176 S  0.0  2.3   0:00.00 go-carbon
14945 root      20   0 3671m 2.9g 4176 S  0.0  2.3   0:00.00 go-carbon
14946 root      20   0 3671m 2.9g 4176 S  0.0  2.3  20:39.26 go-carbon
14949 root      20   0 3671m 2.9g 4176 S  0.0  2.3  19:01.72 go-carbon
14954 root      20   0 3671m 2.9g 4176 S  0.0  2.3  19:50.05 go-carbon
14955 root      20   0 3671m 2.9g 4176 S  0.0  2.3  14:40.24 go-carbon
15466 root      20   0 3671m 2.9g 4176 S  0.0  2.3  14:47.81 go-carbon

New puppet module

Hi
We use go-carbon in production and it enabled us to scale our graphite setup far beyond what we imagined.
We decided to contribute back and open source our puppet module which provisions multiple go-carbon instances on a machine.
We thought you should know.. :-)

https://github.com/similarweb/puppet-go_carbon

Update operations limited to max 50k

With normal traffic i get max 50k on update operation whatever i set in go-carbon config.

There are two situations when it is much higher:

when i set max updates to unlimited
when i stop go-carbon and it's start flushing to drives

I think there is problem with throttling in go-carbon. Any ideas ?

On unlimited setting drives handling traffic with 200k+ update with no problem, but i use too much system cpu, drives lifetime is going down on SSD fast and i would like to have option of tuning IOPS<->Cache usage with more precision.

[common]
user = "carbon"
logfile = "/var/log/go-carbon/go-carbon.log"
log-level = "debug"
graph-prefix = "carbon.agents.{host}."
max-cpu = 15
metric-interval = "10s"

[whisper]
data-dir = "/opt/graphite/storage/whisper/"
schemas-file = "/etc/go-carbon/storage-schemas.conf"
aggregation-file = "/etc/go-carbon/storage-aggregation.conf"
workers = 32
max-updates-per-second = 75000
enabled = true

[cache]
max-size = 10000000
input-buffer = 500000

[udp]
listen = ":2003"
enabled = true
log-incomplete = true

[tcp]
listen = ":2003"
enabled = true

[pickle]
listen = ":2005"
enabled = false

[carbonlink]
listen = "127.0.0.1:7002"
enabled = true
read-timeout = "30s"
query-timeout = "1s"

[pprof]
listen = "localhost:7007"
enabled = false

[2016-01-15 12:34:14] I [persister] doCheckpoint() {commitedPoints=324155, created=0, updateOperations=39988}
[2016-01-15 12:34:24] I [persister] doCheckpoint() {commitedPoints=694713, created=0, updateOperations=18599}
[2016-01-15 12:34:34] I [persister] doCheckpoint() {commitedPoints=1569877, created=0, updateOperations=33807}
[2016-01-15 12:34:44] I [persister] doCheckpoint() {commitedPoints=1618408, created=0, updateOperations=34857}
[2016-01-15 12:34:54] I [persister] doCheckpoint() {commitedPoints=1157205, created=0, updateOperations=40795}
[2016-01-15 12:35:04] I [persister] doCheckpoint() {commitedPoints=1076430, created=0, updateOperations=44264}
[2016-01-15 12:35:14] I [persister] doCheckpoint() {commitedPoints=994659, created=0, updateOperations=46762}
[2016-01-15 12:35:24] I [persister] doCheckpoint() {commitedPoints=965929, created=0, updateOperations=46725}
[2016-01-15 12:35:34] I [persister] doCheckpoint() {commitedPoints=550507, created=0, updateOperations=24489}
[2016-01-15 12:35:44] I [persister] doCheckpoint() {commitedPoints=1013312, created=0, updateOperations=38711}
[2016-01-15 12:35:54] I [persister] doCheckpoint() {commitedPoints=1003661, created=0, updateOperations=38305}
[2016-01-15 12:36:04] I [persister] doCheckpoint() {commitedPoints=786854, created=0, updateOperations=39359}
[2016-01-15 12:36:14] I [persister] doCheckpoint() {commitedPoints=730721, created=0, updateOperations=41951}
[2016-01-15 12:37:26] I [persister] doCheckpoint() {commitedPoints=305571, created=0, updateOperations=31096}
[2016-01-15 12:37:36] I [persister] doCheckpoint() {commitedPoints=1228619, created=0, updateOperations=38687}
[2016-01-15 12:37:46] I [persister] doCheckpoint() {commitedPoints=1439732, created=0, updateOperations=26298}
[2016-01-15 12:37:56] I [persister] doCheckpoint() {commitedPoints=1493276, created=0, updateOperations=24295}
[2016-01-15 12:38:06] I [persister] doCheckpoint() {commitedPoints=1835976, created=0, updateOperations=33459}
[2016-01-15 12:38:16] I [persister] doCheckpoint() {commitedPoints=1144794, created=0, updateOperations=25530}
[2016-01-15 12:38:26] I [persister] doCheckpoint() {commitedPoints=1005914, created=0, updateOperations=25490}
[2016-01-15 12:38:36] I [persister] doCheckpoint() {commitedPoints=1279118, created=0, updateOperations=40149}
[2016-01-15 12:38:46] I [persister] doCheckpoint() {commitedPoints=1217856, created=0, updateOperations=43423}
[2016-01-15 12:38:56] I [persister] doCheckpoint() {commitedPoints=1009972, created=0, updateOperations=45512}
[2016-01-15 12:39:06] I [persister] doCheckpoint() {commitedPoints=947630, created=0, updateOperations=41562}
[2016-01-15 12:39:16] I [persister] doCheckpoint() {commitedPoints=771803, created=0, updateOperations=32007}
[2016-01-15 12:39:26] I [persister] doCheckpoint() {commitedPoints=887938, created=0, updateOperations=31423}

Many strange creates when flushing/USR2

We have big traffic in graphite creates are very delayed on go-carbon with such traffic.

When we stop or USR2 on go-carbon then, a huge number of creates flushing on drives. I think it is related to #32

Most of this creates are empty or have 2-10 data updates and then nothing - null values

Is there any setting i miss ?

Install issues

Hello!

Thanks for nice toolkit!

I have installed it on Ubuntu 14.04 and got following message:

sudo dpkg -i go-carbon_0.6_amd64.deb 
Выбор ранее не выбранного пакета go-carbon.
(Чтение базы данных … на данный момент установлен 85491 файл и каталог.)
Preparing to unpack go-carbon_0.6_amd64.deb ...
Unpacking go-carbon (0.6) ...
Настраивается пакет go-carbon (0.6) …
2015/11/23 23:03:23 open /data/graphite/schemas: no such file or directory
Processing triggers for ureadahead (0.100.0-16) ..

Looks like you need to change path to standard :)

[Feature Request] whitelist.conf / blacklist.conf / query-bulk support

query-bulk

# It's more effective, but python-carbon 0.9.13 (or latest from 0.9.x branch) is required
# See https://github.com/graphite-project/carbon/pull/132 for details
#CARBONLINK_QUERY_BULK = True

whitelist.conf

# This file takes a single regular expression per line
# If USE_WHITELIST is set to True in carbon.conf, only metrics received which
# match one of these expressions will be persisted. If this file is empty or
# missing, all metrics will pass through.
# This file is reloaded automatically when changes are made
.*

blacklist.conf

# This file takes a single regular expression per line
# If USE_WHITELIST is set to True in carbon.conf, any metrics received which
# match one of these expressions will be dropped
# This file is reloaded automatically when changes are made
^some\.noisy\.metric\.prefix\..*

Queries on go carbon

Yesterday I installed go carbon, it works like charm! I was using 10 cache process to handle around 5.5M metrics per minutes, now its working with one process - "go-carbon".
I have few queries, if anyone can help me on this, it would be great help:

Is there any detailed documentation of go-carbon.
what does all internal metrics means - again related to 1.
Is there any way to track "Dropped Metric"

Thanks,
Ravi Bhushan

go-graphite / go-carbon Goto Github PK

go-carbon's People

Contributors

Stargazers

Watchers

Forkers

go-carbon's Issues

Recommend Projects

Recommend Topics

Recommend Org