markchadwick/graphite-relay

A fast Graphite relay

Graphite Relay

A fast(er) Graphite relay written with Netty. This will read updates in the line-oriented format (metric value timestamp) and write Pickle format to the backends.

Running the Relay

java -jar graphite-relay-0.1.jar -c /path/to/relay.properites

Backend Strategies

A backend strategy will pick which Graphite backends to send a given metric to. The include ones are:

• Broadcast - Broadcast each metric to all backends
• ConsistentHash - Use a simple consistent hash to send an update to a backend with a minimal change in (metric -> backend) mapping on backend addition or deletion.
• RoundRobin - Cycle through the backends sending a batch of metrics to one at a time (Not reccomended as this will not play nicely with other Graphite Tools).

Overflow Handlers

When a backend goes down, this relay will buffer up messages to a point. After that point, it will dump updates to an OverflowHandler to ensure we don't blow the heap. The OverflowHandler should do something with an update toher than keep it in memory. The included implementations are:

• BitchingOverflowHandler - Simply log ever so often to let someone know that not all metrics are making it to backends.
• LoggingOverflowHandler - Stream the metrics to a rolling, gzipped set of files on disk which can be replayed with netcat (or similar) at a later point.

Config Properties

There are a number of variables you can set in the config file. Popular ones are set below. If a required parameter is missing it will manifest itself (for the moment) with Guice vomiting a stack trace on you while trying to start the program.

relay.backends

Graphite backends which will recieve messages in the normal Graphite Pickle format. This should be a line-delimited list of host:port pairs. For example:

relay.backends: \
    localhost:1234 \
    localhost:1235 \
    localhost:1236 \
    localhost:1237

relay.hostbuffer

Number of updates to buffer for each host in the event that it goes down

relay.port

Default port to listen on. This port will expect line-oriented Graphite update metrics.

relay.reconnect

Number of seconds to sleep before trying to reconnect to a disconnected backend.

relay.backendstrategy

Backend Strategy to use for finding a backend for each metric. Default available values are:

• graphite.relay.backend.strategy.Broadcast
• graphite.relay.backend.strategy.ConsistentHash
• graphite.relay.backend.strategy.RoundRobin You may also set to the FQCN of any other BackendStrategy in the CLASSPATH. If using the ConsistentHash strategy, you will also have to set hash.replicas in the config. A reasonable default value is 10.

relay.overflowhandler

Handler which will recieve updates that no backend is available to handle. This is generally because it is unavailable or overwhelmed. Default available values are:

• graphite.relay.overflow.BitchingOverflowHandler
• graphite.relay.overflow.LoggingOverflowHandler Like relay.backendstrategy, the FQCN of any other OverflowHandler in the CLASSPATH is just as valid. If using the LoggingOverflowHandler, you must set overflow.directory in the configuration to a directory that the current user has permission to create.

A complete config file might be as follows:

relay.hostbuffer:   1000
relay.port:         2002
relay.reconnect:    2

relay.backendstrategy: graphite.relay.backend.strategy.ConsistentHash
hash.replicas: 20

relay.overflowhandler: graphite.relay.overflow.LoggingOverflowHandler
overflow.directory:    /mnt/overflow/

relay.backends: \
    localhost:1234 \
    localhost:1235 \
    localhost:1236 \
    localhost:1237

Pickle Format

The included Pickle formatting is very rudimentary and only groks the data type that the Graphite backends expect. Because of its simple format, this will actually use a very early Pickle encoding, which cPickle and friends can grok, but may not be most effecient.

graphite-relay's Issues

Working with Graphite 0.9.11 ?

Hi Mark,

Trying to test your relay against our installation of Graphite 0.9.11 - relay and 4 caches with consistent hashing.
Found everything "working" but graphs on Graphite-web looks totally wrong - like we loosing all caches but one:

Checked that's not true - all 4 caches got updates, and all updates went to disk.
So, my idea was that hashing algorithm in your implementation differs than what Graphite use, and I found that you use just simple CRC32:
https://github.com/markchadwick/graphite-relay/blob/master/src/main/scala/backend/strategy/ConsistentHash.scala#L39-43
but Graphite use int( md5hex() [:4])
https://github.com/graphite-project/carbon/blob/master/lib/carbon/hashing.py#L16-19
https://github.com/graphite-project/graphite-web/blob/master/webapp/graphite/render/hashing.py#L66-69
Could you please tell me if I'm right and providing proper hashing will help? ('cause I'm not very good at scala and I want to spend my time on this patch only if I'm thinking right)

Thanks!

Compiling the graphite-relay

Hi Mark,

Sorry for the dumb question. If I have to compile the source code, do I need to compile it in openjdk-7? Can you point me to some documentation for the same for me to do the same?

If the documentation is in progress, is there a jar file uploaded somewhere that I can use? I would be happy to help you with the documentation.

Add a license?

I would like to use some of your code to implement some load tests against Graphite using Iago (https://github.com/twitter/iago) but wont be able to publish them unless you declare a license. Without specifying, the copyright defaults to 'all rights reserved'.

Can you take a moment to drop a LICENSE file you like in the repo? It'd be much appreciated :)

Too many CLOSE_WAIT connections

After some running (20 minutes) on our test environment I fount that relay open tons of connections to client:
root@graph:/# netstat -apn | grep 25782 | grep ESTA | wc -l
1297
root@graph:/# netstat -apn | grep 25782 | grep CLOS | wc -l
2669
As far as I understand these connections are holded by Netty. How can I mitigate this?

Thanks!

Startup errors

Hello,

I would like to use this graphite-relay app with one of my projects, but the issue is that I have not been able to lunch the relay server (graphite-relay-0.2.7.jar).

Is it necessary to have any additional jar in the classpath?

java -version

java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)

java -jar graphite-relay-0.2.7.jar -c graphite-relay.properties

Loading graphite-relay.properties
2012-06-05 13:00:22,741 [main ] INFO Main - Backend Strategy: class graphite.relay.backend.strategy.ConsistentHash
2012-06-05 13:00:22,742 [main ] INFO Main - Overflow Handler: class graphite.relay.overflow.BitchingOverflowHandler
Exception in thread "main" com.google.inject.ConfigurationException: Guice configuration errors:

1. No implementation for java.lang.String annotated with @com.google.inject.name.Named(value=aggregation.config) was bound.
   while locating java.lang.String annotated with @com.google.inject.name.Named(value=aggregation.config)
   for parameter 2 at graphite.relay.aggregation.Aggregator.(Aggregator.scala:19)
   while locating graphite.relay.aggregation.Aggregator

1 error
at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:1004)
at com.google.inject.internal.InjectorImpl.getProvider(InjectorImpl.java:961)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
at graphite.relay.Main$.main(Main.scala:43)
at graphite.relay.Main.main(Main.scala)

Thanks in advance for any help.