sivy/node-statsd

node.js client for Etsy'd StatsD server

node-statsd

A node.js client for Etsy's StatsD server.

This client will let you fire stats at your StatsD server from a node.js application.

node-statsd Runs and is tested on Node 0.6+ on all *nix platforms and 0.8+ on all platforms including Windows.

Installation

$ npm install node-statsd

Usage

All initialization parameters are optional.

Parameters (specified as an options hash):

• host: The host to send stats to default: localhost
• port: The port to send stats to default: 8125
• prefix: What to prefix each stat name with default: ''
• suffix: What to suffix each stat name with default: ''
• globalize: Expose this StatsD instance globally? default: false
• cacheDns: Cache the initial dns lookup to host default: false
• mock: Create a mock StatsD instance, sending no stats to the server? default: false
• global_tags: Optional tags that will be added to every metric default: []

All StatsD methods have the same API:

• name: Stat name required
• value: Stat value required except in increment/decrement where it defaults to 1/-1 respectively
• sampleRate: Sends only a sample of data to StatsD default: 1
• tags: The Array of tags to add to metrics default: []
• callback: The callback to execute once the metric has been sent

If an array is specified as the name parameter each item in that array will be sent along with the specified value.

  var StatsD = require('node-statsd'),
      client = new StatsD();

  // Timing: sends a timing command with the specified milliseconds
  client.timing('response_time', 42);

  // Increment: Increments a stat by a value (default is 1)
  client.increment('my_counter');

  // Decrement: Decrements a stat by a value (default is -1)
  client.decrement('my_counter');

  // Histogram: send data for histogram stat
  client.histogram('my_histogram', 42);

  // Gauge: Gauge a stat by a specified amount
  client.gauge('my_gauge', 123.45);

  // Set: Counts unique occurrences of a stat (alias of unique)
  client.set('my_unique', 'foobar');
  client.unique('my_unique', 'foobarbaz');

  // Incrementing multiple items
  client.increment(['these', 'are', 'different', 'stats']);

  // Sampling, this will sample 25% of the time the StatsD Daemon will compensate for sampling
  client.increment('my_counter', 1, 0.25);

  // Tags, this will add user-defined tags to the data
  client.histogram('my_histogram', 42, ['foo', 'bar']);

  // Using the callback
  client.set(['foo', 'bar'], 42, function(error, bytes){
    //this only gets called once after all messages have been sent
    if(error){
      console.error('Oh noes! There was an error:', error);
    } else {
      console.log('Successfully sent', bytes, 'bytes');
    }
  });

  // Sampling, tags and callback are optional and could be used in any combination
  client.histogram('my_histogram', 42, 0.25); // 25% Sample Rate
  client.histogram('my_histogram', 42, ['tag']); // User-defined tag
  client.histogram('my_histogram', 42, next); // Callback
  client.histogram('my_histogram', 42, 0.25, ['tag']);
  client.histogram('my_histogram', 42, 0.25, next);
  client.histogram('my_histogram', 42, ['tag'], next);
  client.histogram('my_histogram', 42, 0.25, ['tag'], next);

Errors

In the event that there is a socket error, node-statsd will allow this error to bubble up. If you would like to catch the errors, just attach a listener to the socket property on the instance.

client.socket.on('error', function(error) {
  return console.error("Error in socket: ", error);
});

If you want to catch errors in sending a message then use the callback provided.

License

node-statsd is licensed under the MIT license.

node-statsd's Issues

allow disabling statsd

use case:
a node app implements statsd metrics sending using this library.
user of this node app wants to configure his app so that it doesn't do any statsd sending.

i see 3 ways to implement this:

• everywhere the app does calls on the client, wrap in disabled/enabled check (ugly)
• app provides wrapper functions around library to wrap all calls in disabled/enabled check (still a bit ugly)
• node-statsd doesn't do any statsd sending if the Client has this.host unset, or empty or null or whatever. (nice!)

the only thing needed to make this work is an edit in the Client.prototype.send function. I'm willing to create a patch for this, but since there's a bunch of socket-related changes pending, I would wait until that settles down.

Forked your work

Would be awesome to just merge it all in, but you seemed unresponsive to merge stuff and I think this was due.

If you feel like merging back, let me know, I can send a pull request and delete my fork from npm.

Multi-metric packets

It seems like it would be feasible to send batched messages through this api as well as one-metric-per udp-packet messages. Would this fit into the current plans?

Avoiding DNS Lookups for localhost on socket.send

Greetings! I recently noticed that nodes udp library does a dns lookup on every socket.send call. It looks like newer versions of node allow callers to specify a custom lookup function on socket creation dgram.createSocket. But older versions don't.

Would it be possible to introduce a change to allow callers to specify a createSocket function as part of the statsd client arguments. This approach was taken in order to enable backwards compatibility with wayyy older versions of node (ie v0.10).

I appreciate your time!

danny

#76

node-statsd needs a new maintainer

Hi folks:

This project needs a maintainer - I was the original author but I don't use node.js on a regular basis and my knowledge was never that deep. There are some great devs here, and I'm hoping one of you might like to take on the project.

@rcrowley, @lloyd, @mojodna, @msiebuhr, @salty-horse, @CodeFridge, @kastner ...

If one of you guys is interested in taking over maintainership, or knows someone who would be, get in touch and I'll hand over the reigns. Once we work out who (or group of whos) is going to take over, you can update the README with a note as to the new "canonical" repo, and give me a pull request, and I'll merge it here.

Thanks for participating and helping to make node-statsd a better tool than it was when I started! :-)

--Steve

how to add mutli entry using line protocol in statsd

Can you let me know how to add
cpu,host=serverA,region=us_west value=0.64

in statsd timer

  var StatsD = require('node-statsd'),
      client = new StatsD();

  // Timing: sends a timing command with the specified milliseconds
  client.timing('response_time', 42);

node_modules checked into package

The directory node_modules shouldn't be in the repository (the package mersenne was added in f379633). Ideally, it should be a dependency of the package.

(Or has the mersenne-package been hacked to work for node-statsd?)

ENOTFOUND on localhost

Hi,

I wish I could shine some more light on this, and was perhaps wondering if you've ever seen this before rather than being able to report a specific issue.

We have the following code in our websocket server:

 var StatsD = require('node-statsd').StatsD;
    var statsdClient = new StatsD({ prefix: config.get('stats:statsd:prefix') });

    statsdClient.socket.on('error', function(error) {
      return emergencyLog("Error in statsd socket: " + error, { exception: error });
    });

We've seen it happen on two occasions now where our log files will just start spitting this out continuously.

Error: getaddrinfo ENOTFOUND { exception: { [Error: getaddrinfo ENOTFOUND] code: 'ENOTFOUND', errno: 'ENOTFOUND', syscall: 'getaddrinfo' } }

What doesn't make any sense is that our understanding of ENOTFOUND is DNS level. If there was something iffy with the statsd server, we'd probably get ECONNREFUSED. Restarting the service seems to make the problem magically go away.

Less than helpful, I know but was wondering if you had any theories.

(Feel free to discuss at https://gitter.im/gitterHQ/nodejs).

increment(0) == increment(1)

I know I shouldn't have called increment(0) but let me tell you it took a long time to find that bug !

Because increment (and decrement) compare delta to 0/null and not to undefined

StatsDClient.prototype.increment = function (name, delta) {
    this.counter(name, Math.abs(delta || 1));
};

could become :

StatsDClient.prototype.increment = function (name, delta) {
    this.counter(name, Math.abs(delta!=undefined?delta: 1));
};

or

StatsDClient.prototype.increment = function (name, delta) {
    if (delta==0) then return; // no op
    this.counter(name, Math.abs(delta || 1));
};

Why was it hard to figure it out:

• my code could have been the problem (triple checked that)
• custom statsd -> influxdb connector (which I thought was faulty)
• influxdb could have been the problem too (checked that)
• statsd could have been the problem (checked that too)

I think you could document that special case... or allow delta == 0

Many thanks for all your hard work anyway!

errors when installing node-statds

On the latest version, I'm getting the following errors when trying to npm install node-statsd (using node 0.10.20, npm 1.3.11)

npm WARN package.json [email protected] No repository field.
npm WARN package.json [email protected] 'repositories' (plural) Not supported.
npm WARN package.json Please pick one as the 'repository' field
npm http GET https://github.com/downloads/lloyd/node-statsd/0509f85.tgz
npm http 200 https://github.com/downloads/lloyd/node-statsd/0509f85.tgz
npm WARN optional dep failed, continuing [email protected]

even though it fails the optional dep. no node-statsd dir exists in the ./node-modules dir after npm finishes

exception handling

I would try to implement the following exception policy in node-statsd:

• exceptions "bubble up" into the app that uses this library
• we don't log or print to console any errors ourself, it's the toplevel app that decides how to log/write to console.
• we document which exceptions can be raised, and where.
  I'm thinking of the following issues:
• dns resolve (exception EADDRINFO)
• host unreachable, connection refused (these don't give any exception, but that's ok: it's a udp socket anyway)
• any other exception that might get raised?? I surely must be missing some, because at least require('dgram').createSocket('udp4') can raise errors, and socket.send() as well.

I'm new to node.js exception handling, so any input is welcome. For now I'm just tinkering and trying to find a workable solution, but I haven't figured it out yet. cc @msiebuhr

F

Leaks file descriptors

See msilvey@0ade323, https://github.com/37signals/node-statsd/commit/d35ad6127e5cf88c0c859fa4f9e641cc31c8fe13 and other forks.

Additionally, one socket can be created upfront and re-used for the life of the process.

update npm package

Any plans to do so? I'm missing the gauge function for example.

Best regards

Philipp

Tag recent releases

the npm registry knows about releases up to 0.0.7 but the latest published git tag os 0.0.2

Publish 0.1.0 to npm

Please publish the latest version 0.1.0 to npm.

merging back with other clients?

Any chance I help with merging with the other 4 modules I found?

https://github.com/msiebuhr/node-statsd-client
https://github.com/sivy/node-statsd
https://github.com/spreaker/nodejs-statsd-client
https://github.com/dscape/lynx

Npm publish v0.1.1

Please publish v0.1.1 to npm, it resolves some issues in node v0.12.x

Thank you!

fix gauges spelling in readme

The spelling of gauges is currently quite inconsistent in the readme. Sometimes it's written as gauges and sometimes as guages.

Can I use `.send()` to send raw data?

I needs an opportunity to send raw text, can I use method .send() for it?
Is so, I can add it to docs, otherwise I can make a pull request to add method .raw() for example.

intercept send out metrics

Hey,

I would like to intercept the outgoing metrics and for example prevent some of them to be really send. What do you think, would it be ok to directly intercept the internal send function or is that function rather unstable and might change over time ?

My use case is the following: I would like to write a hapi plugin that exposes the statsd client and automatically tracks route calls and timings, but also gives the user the possibility to send custom metrics. Now these statsd metrics could be send to aws cloudwatch and i would like to be able to limit the amount of different metrics that are send out.

So should I just intercept send or would it be possible to add some interception api ? I would like to prepare a PR if you are interested into that ?

Process never exits after statsd use

The code

'use strict'

setInterval(() => {
    console.log('Still alive')
}, 1000).unref()

setTimeout(() => {
    console.log('Done')
}, 500)

prints "Done" then exits.

$ node index.js
Done
$

However, once I introduce a statsd client, the process never exists.

'use strict'

const StatsD = require('node-statsd')
const statsd = new StatsD()

setInterval(() => {
    console.log('Still alive')
}, 1000).unref()

setTimeout(() => {
    statsd.increment('my_counter')
    console.log('Done')
}, 500)

The output is

$ node index.js
Done
Still alive
Still alive
Still alive
Still alive
Still alive
Still alive
Still alive
Still alive
Still alive
...

Is there a way to tell statsd that it should cleanup and not keep the node process alive?

How can I send the metrics to all the host in docker container?

I am currently running the node-statsd in a docker container. I want to pass the metrics from the container in which the application is running to the container which is listening on localhost:8125. How can I specify the host to do the same?

Problem with fs.appendFile with Statd Client ?

This one looks so strange to me. I'm assuming this problem is because of StatD because once I remove it from code, it is working super fine. Here is the details about the issue:

I've Kafka Data floating here and there. I've node js client which consumes the data from Kafka and store it locally for 100 ms. Then it flush to disk at every 100 ms.

The code will be something similar like this:

const MainApp = require('./MainApp');

const fs = require('fs');
const path = require('path');
const uuid = require('uuid');


const zkUrl = process.env.ZK_URL;
const topics = process.env.SYNC_TOPICS_TO_CONSUME.split(' ');

const consumerOptions = {
  zookeeperUrl: zkUrl,
  groupId: process.env.KAFKA_SYNCER_GROUP_ID,
  serverPort: 3043,
  threadCount: parseInt(process.env.KAFKA_SYNCER_WORKER_COUNT || 1, 10),
  properties: { 'rebalance.max.retries': '3' }
};

function mkdirp(_filepath) {
  const dirname = path.dirname(_filepath);
  if (!fs.existsSync(dirname)) {
    mkdirp(dirname);
  }
  fs.mkdirSync(_filepath);
}


class Syncer extends MainApp {
  constructor(options) {
    if (!options) optiosn = {};
    super();
    Object.defineProperties(this, {
      bigData: {
        value: {},
        writable: true,
        enumerable: true
      }
    });

    if (options.autoSyncDb) {
      const self = this;
      const interval = parseInt(options.autoSyncDbInterval, 10) || 100;
      setTimeout(function _orbSyncDb() {
        self.syncDb();
        setTimeout(_orbSyncDb, interval);
      });
    }
  }

  pushMessage(message, partition) {
    if (this.bigData[partition] === undefined) {
      this.bigData[partition] = [message];
    } else {
      this.bigData[partition].push(message);
    }
  }

  syncDb() {
    this._bigDataCopy = this.bigData;
    this.bigData = {};

    Object.keys(this._bigDataCopy).forEach((_partition) => {
      const days = this._bigDataCopy[_partition];

      this.fileName = 'events.json';
      this.fileMessages = {};

      days.forEach((day) => {
        
        fs.readdirSync(filePath).forEach((file) => {
          if (file.indexOf('events.json') > 0) {
            const fileStats = fs.statSync(filePath + file);
            if (!fullfilePath && ((fileStats.size / 1000000) < 500)) {
              fullfilePath = filePath + file;
            }
          }
        });

        if (!fullfilePath) {
          fullfilePath = filePath + uuid() + this.fileName;
        }

        fs.appendFile(
          fullfilePath,
          day.map((_msg) => { // eslint-disable-line
            return JSON.stringify(_msg);
          }).join('\n') + '\n',
          (err) => {
            if (err) MainApp.log.error(err, [filePath, this.fileName]);
            console.log(
              "Flushed to disk: " + (this.fileMessages[msgDay] || []).length
              + " Messages,");
          }
        );
      });
    });
  }

  forceFlush() {
    this.syncDb();
  }
}

const syncer = new Syncer({
  autoSyncSchema: true,
  autoSyncSchemaInterval: 30000,
  autoSyncDb: true,
  autoSyncDbInterval: 100
});

class MessageParser extends MainApp {
  constructor(message, topic) {
    super();
    this.topic = topic;
    if (typeof message === 'string') {
      try {
        this.message = JSON.parse(message);
      } catch (e) {
        this.message = message;
        this.log.error(e, [message]);
      }
    } else {
      this.message = message;
    }
  }

  getDay (){
    return this.message.timestamp.split('T')[0];
  }
}

const onMessage = function onKafkaMessage(message, topic, callback) {
  this.event = new MessageParser(message, topic);
  syncer.pushMessage(this.event.getDay(), this.event.message);
  process.nextTick(callback);
};


MainApp.consumer.consume(topics, onMessage, consumerOptions);

process.on('SIGINT', () => {
  if (MainApp.consumer.doesExist()) MainApp.consumer.stop();
  syncer.forceFlush();
  MainApp.log.info('Consumer Stopped!');
  process.exit();
});

process.on('uncaughtException', (err) => {
  MainApp.log.error(err);
  process.exit();
});

The above one is working super fine., and the tough part starts here. When I made changes and added statsD, appendSync is not working at all,

here is the code after changes

const MainApp = require('./MainApp');

const fs = require('fs');
const path = require('path');
const uuid = require('uuid');


const statsdOptions = {
  host: process.env.STATSD_HOST,
  port: process.env.STATSD_PORT,
  prefix: 'vtap.syncer.',
  globalize: true
};

Vtap.statsD(statsdOptions);

const zkUrl = process.env.ZK_URL;
const topics = process.env.SYNC_TOPICS_TO_CONSUME.split(' ');

const consumerOptions = {
  zookeeperUrl: zkUrl,
  groupId: process.env.KAFKA_SYNCER_GROUP_ID,
  serverPort: 3043,
  threadCount: parseInt(process.env.KAFKA_SYNCER_WORKER_COUNT || 1, 10),
  properties: { 'rebalance.max.retries': '3' }
};

function mkdirp(_filepath) {
  const dirname = path.dirname(_filepath);
  if (!fs.existsSync(dirname)) {
    mkdirp(dirname);
  }
  fs.mkdirSync(_filepath);
}


class Syncer extends MainApp {
  constructor(options) {
    if (!options) optiosn = {};
    super();
    Object.defineProperties(this, {
      bigData: {
        value: {},
        writable: true,
        enumerable: true
      }
    });

    if (options.autoSyncDb) {
      const self = this;
      const interval = parseInt(options.autoSyncDbInterval, 10) || 100;
      setTimeout(function _orbSyncDb() {
        self.syncDb();
        setTimeout(_orbSyncDb, interval);
      });
    }
  }

  pushMessage(message, partition) {
    if (this.bigData[partition] === undefined) {
      this.bigData[partition] = [message];
    } else {
      this.bigData[partition].push(message);
    }
  }

  syncDb() {
    this._bigDataCopy = this.bigData;
    this.bigData = {};

    Object.keys(this._bigDataCopy).forEach((_partition) => {
      const days = this._bigDataCopy[_partition];

      this.fileName = 'events.json';
      this.fileMessages = {};

      days.forEach((day) => {
        
        fs.readdirSync(filePath).forEach((file) => {
          if (file.indexOf('events.json') > 0) {
            const fileStats = fs.statSync(filePath + file);
            if (!fullfilePath && ((fileStats.size / 1000000) < 500)) {
              fullfilePath = filePath + file;
            }
          }
        });

        if (!fullfilePath) {
          fullfilePath = filePath + uuid() + this.fileName;
        }

        fs.appendFile(
          fullfilePath,
          day.map((_msg) => { // eslint-disable-line
            return JSON.stringify(_msg);
          }).join('\n') + '\n',
          (err) => {
            if (err) {
            MainApp.log.error(err, [filePath, this.fileName]);
            }
            else {
            this.flushedMsgCount = (day || []).length;
            console.log(`Flushed to disk: ${day} Messages.`);
            }
            
          }
        );
      });
    });
  }

  forceFlush() {
    this.syncDb();
  }
}

const syncer = new Syncer({
  autoSyncSchema: true,
  autoSyncSchemaInterval: 30000,
  autoSyncDb: true,
  autoSyncDbInterval: 100
});

class MessageParser extends MainApp {
  constructor(message, topic) {
    super();
    this.topic = topic;
    if (typeof message === 'string') {
      try {
        this.message = JSON.parse(message);
      } catch (e) {
        this.message = message;
        this.log.error(e, [message]);
      }
    } else {
      this.message = message;
    }
  }

  getDay (){
    return this.message.timestamp.split('T')[0];
  }
}

const onMessage = function onKafkaMessage(message, topic, callback) {
  this.event = new MessageParser(message, topic);
  syncer.pushMessage(this.event.getDay(), this.event.message);
  process.nextTick(callback);
};


MainApp.consumer.consume(topics, onMessage, consumerOptions);

process.on('SIGINT', () => {
  if (MainApp.consumer.doesExist()) MainApp.consumer.stop();
  syncer.forceFlush();
  MainApp.log.info('Consumer Stopped!');
  process.exit();
});

process.on('uncaughtException', (err) => {
  MainApp.log.error(err);
  process.exit();
});

I don't understand what causing the issue. I don't see any error message, any console message at all. To debug I added console log every where,

Till FileFullPath it is working, after that no logs are created. Any idea why it is failing to write to disk ?

Sending tags to StatsD give a "Bad line" error

If you specify tags and you send the metric agains the StatsD server instead of the DataDog-StatsD server you get the following error:

28 May 10:02:26 - DEBUG: Bad line: 15,c,#a in msg "example.tags.count:15|c|#a"
28 May 10:02:40 - DEBUG: Bad line: 15,c,#a=1 in msg "example.tags.count:15|c|#a=1"
28 May 10:02:43 - DEBUG: Bad line: 15,c,#a in msg "example.tags.count:15|c|#a:1"
28 May 10:02:43 - DEBUG: Bad line: 1 in msg "example.tags.count:15|c|#a:1"

non-exclusive dgram bind can cause cluster exceptions

So some versions of node have this bug: nodejs/node-v0.x-archive#9261

It is triggered when a clustered node process uses client datagram sockets.

There's a workaround to bind your client sockets with {exclusive: true} to opt them out of cluster sharing (details in the bug report).

Wonder if you'd consider adding the workaround to this library. I'm seeing it manifest in the node-dogstatsd fork of this module, but can't file issues on the fork, so I'm hoping to get it patched upstream then get the node-dogstatsd maintainer to pull the fix downstream.

Don't publish tests to npm

Pretty please? :) Can you add a .npmignore file and have it exclude the test directory from being published?

dnsCache != cacheDns

The README.md file says that the parameter for DNS caching is "dnsCache", but the code recognizes "cacheDns" instead:
https://github.com/sivy/node-statsd/blob/master/lib/statsd.js#L39

Omitting callback may throw uncaught exception

When not passing a callback to the sending functions (any), DNS errors would result in an exception being thrown like this:

events.js:66
        throw arguments[1]; // Unhandled 'error' event
                       ^
Error: getaddrinfo EADDRINFO
    at errnoException (dns.js:31:11)
    at Object.onanswer [as oncomplete] (dns.js:123:16)

See CartoDB/Windshaft-cartodb#180

Data not saved in graphite

When I try to use gauge for a graphite server, I don't get any errors, but the data is not saved.I use graphite:latest from docker, node-statsd version 0.1.1. Code snippet:

graphite = new StatsD({
    host: GRAFANA_SERVER,
    port: GRAFANA_PORT,
    mock: false
});

graphite.socket.on('error', function(error) {
    return console.error('Error', error);
});

graphite.gauge('gauge.two', 2, undefined, undefined, function(err, bytesSent) {
    console.log('Finished gauge', err, bytesSent); // bytesSent=13
    graphite.close();
});

callback usage

If a callback is not provided is the call to the client synchronous?

node-statsd binds to all hosts by default

When creating nodejs' dgram.Socket it is bound to all hosts by default. Is this intentional?

For public servers it might be a good idea to only bind to specific interfaces (e.g. localhost); this can be modified by calling Socket.bind.

Is sampleRate required for .increment?

> var StatsD = require('node-statsd').StatsD;
undefined
> var client = new StatsD();
undefined
> client.increment('foo', 1, function() { console.log('fin'); })
undefined
> client.increment('foo', 1, 1, function() { console.log('fin'); })
undefined
> fin

undefined
>

I waited a few seconds after the first call to increment, and callback was never invoked.

Using 0.0.7

I'd be happy to create a PR to update README with more detailed info about which args are optional/required for each method exposed by StatsD, if you're interested.