hubspot/buckyserver

Node server that receives metric data over HTTP & forwards to your service of choice

buckyserver's Issues

On a page with a high amount of Ajax calls (for examples) Lynx fails to send a correct UDP packet to statsd

Resulting in many drops metrics. See dscape/lynx#4

If Lynx doesn't provide a fix, maybe Bucky can forgo batch export or make a few lighter calls?

multi-metric packets

We currently push a lot of things to statds with multi-metric packets

some.metric.total:33|c\nsome.metric.other.total:2|c\nsome.metric.more.total:77|c\nsome.metric.summore.total:77|c

I have confirmed with tcpdump that multi-metric submissions are not making it through to statsd but single metrics are. Is there any formatting or way to get multi-metric submission through the BuckyServer app and forwarded on to statds?

Compatibility with Hosted Graphite

Great project! I'd like to get this working with HostedGraphite.com
Looks like i need to prefix the metrics with my API Key. Where in the code would you recommend doing this?

Support a configurable CORS origin

The current BuckyServer allows anyone to post metrics, enabling anyone on the internet to spam new keys which can make a mess of a statsd/Graphite server.

https://github.com/HubSpot/BuckyServer/blob/master/server.coffee#L30

It'd be great to have the ability to specify specific Access-Control-Allow-Origin headers, perhaps right around

https://github.com/HubSpot/BuckyServer/blob/master/config/default.yaml#L3

Support whitelisting of keys

BuckyServer currently provides no way to limit the number/volume of keys, enabling anyone on the internet to spam new keys which can make a mess of a statsd/Graphite server.

Optionally whitelisting keys via the config file looks like it would be a pretty straightforward way to fix this.

Running BuckyServer on a Windows machine

Hi all,
I can't seem to make it run on a Windows machine. My guess is that it was not contemplated. Am I right?

Cheers
Ricardo

Dockerise Bucky Server

Example

FROM node

# a hack to fix: EXDEV: cross-device link not permitted, rename
RUN cd $(npm root -g)/npm \
&& npm install fs-extra \
&& sed -i -e s/graceful-fs/fs-extra/ -e s/fs.rename/fs.move/ ./lib/utils/rename.js

RUN mkdir -p /opt/app
ADD . /opt/app
WORKDIR /opt/app

RUN npm install

CMD ["node", "start.js"]

EXPOSE 5999

Communication error between Bucky and statsd

I had previously set up Bucky Server and had everything configured correctly with Graphite and statsd, but had to rebuild the server (Ubuntu 15.10). Since the rebuild, I'm having trouble connecting Bucky and statsd. When refreshing a basic test page with the Bucky client, the server receives the call and tries to send info to statsd, but statsd reports that the messages are bad. The following are logged by Bucky and statsd for a single refresh of the test page. Carbon and statsd are both registering with Graphite and I can register with statsd from bash, so it appears to just be the communication between Bucky and statsd. Any ideas as to what might be happening?

----- logged by Bucky Server:
Server listening on port 5000 in production mode
Collecting index.navigationStart, index.unloadEventStart, index.unloadEventEnd, index.redirectStart, index.redirectEnd, index.fetchStart, index.domainLookupStart, index.domainLookupEnd, index.connectStart, index.connectEnd, index.secureConnectionStart, index.requestStart, index.responseStart, index.responseEnd, index.domLoading, index.domInteractive, index.domContentLoadedEventStart, index.domContentLoadedEventEnd, index.domComplete, index.loadEventStart, index.loadEventEnd for 67.233.128.8
{ [Error: connect ECONNREFUSED]
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect' } 'on socket'

----- logged by statsd:
20 Apr 17:21:55 - [54308] reading config file: /opt/statsd/localConfig.js
20 Apr 17:21:55 - server is up
20 Apr 17:22:45 - DEBUG: Bad line: -1461172959833,ms in msg "index.redirectStart:-1461172959833|ms"
20 Apr 17:22:45 - DEBUG: Bad line: -1461172959833,ms in msg "index.redirectEnd:-1461172959833|ms"
20 Apr 17:22:45 - DEBUG: Bad line: -1461172959833,ms in msg "index.secureConnectionStart:-1461172959833|ms"

Prefix at the server config level

While I can gather metrics automatically by adding data-bucky-page data-bucky-requests to my bucky script include, I cannot specify a prefix there (other than by overloading the page name). Ideally I'd want both a static prefix & the name of the page.

Could I set a prefix at the buckyserver config level (something like 'bucky' for instance)?

Config for appRoot and port is never read from config/default.yml

First of all: kudos to Bucky, I really like the library and the documentation makes it relatively easy to get started. With BuckyServer version 0.2.0 though I noticed a problem with the processing of the config file under config/default.yml: in the function loadApp a config object is passed by as second argument for usage with the load() call. The problem is that the APP_ROOT and the port are also read from this config object, which is wrong - instead they should be read from the config object defined in line 6, which unfortunately has the same name. The result of this is that setting the app root and/or the port in config.yml won't affect Bucky in any way!

CORS issues and 204 response codes from server

I'm having issues with CORS when using my Bucky Server.

Chrome throws a CORS error when the requests are made, but the Bucky console logger shows that the request metrics were indeed posted.

Collecting index.domContentLoadedEventStart, index.domInteractive, index.domLoading, index.responseEnd, index.responseStart, index.requestStart, index.connectEnd, index.connectStart, index.domainLookupEnd, index.domainLookupStart, index.fetchStart, index.unloadEventEnd, index.unloadEventStart, index.navigationStart for 199.180.243.147

Out of curiosity, I decided to use a Chrome plugin called POSTMAN to manually post to the Bucky Server. The server only responds with 204 responses, and never contains any of the CORS response headers that are supposed to be set. So it seems as though something isn't functioning correctly. Requests to the health check route work fine.

Thanks in advance.
-P

Lynx send overload UDP packets

UDP package length is 1472 bytes.
This happened when using send API to send a batch of large metrics.

This should be fixed on Lynx side, but their repository might be inactive.
Is it ok if we change the package to other lynx repository:

'lynx': 'vinhlh/lynx'

Support Node cluster

Currently, Bucky Server only supports single process.
It should support Node cluster to have better performance on multiple core server.

ParsingErrorBuckyServer

Hello,
I am running a CentOS 7 . When trying to run bucky-server i have the following error

/usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:949
    throw new Error("Cannot parse config file: '" + fullFilename + "': " + e3)
          ^
Error: Cannot parse config file: '/usr/lib/node_modules/bucky-server/config/default.yaml': SyntaxError: invalid indentation, got 2.5 instead of 2
    at Config._parseFile (/usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:949:11)
    at /usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:718:25
    at Array.forEach (native)
    at /usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:714:14
    at Array.forEach (native)
    at Config._loadFileConfigs (/usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:713:13)
    at new Config (/usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:118:5)
    at Object.<anonymous> (/usr/lib/node_modules/bucky-server/node_modules/config/lib/config.js:1402:64)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (/usr/lib/node_modules/bucky-server/node_modules/coffee-script/lib/coffee-script/coffee-script.js:211:36)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/usr/lib/node_modules/bucky-server/server.coffee:13:12)
    at Object.<anonymous> (/usr/lib/node_modules/bucky-server/server.coffee:159:4)
    at Module._compile (module.js:456:26)
    at Object.loadFile (/usr/lib/node_modules/bucky-server/node_modules/coffee-script/lib/coffee-script/coffee-script.js:182:19)
    at Module.load (/usr/lib/node_modules/bucky-server/node_modules/coffee-script/lib/coffee-script/coffee-script.js:211:36)
    at Function.Module._load (module.js:312:12)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/usr/lib/node_modules/bucky-server/start.js:4:1)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)
    at startup (node.js:119:16)
    at node.js:935:3

Is it something wrong on my side?