obfuscurity/backstop

HTTP service for submitting metrics to Graphite

backstop's Issues

Backstop should accept compressed payload

Given that carbon is a plain text format, a client should have the option to send data to Backstop compressed, where Backstop decompressed it and sends it on to graphite. This would be particularly interesting for scenarios like HostedGraphite, for example, where I'm sending gobs of data, which could be heavily compressed.

collectdwin to carbon via backstop

Documentation Missing "How to install / Integrate with Graphite"

I didnt find an documentation or an small instructions for install backstop (including libs requirement) step by step for integrate it with graphite.
is it possible to have it into the README ?

Request.body.read always empty?

I realize that I am not posting JSON data in this example, but no matter what I send (valid JSON or not), request.body.read is always empty.

post '/publish/:name' do
      begin
        data = JSON.parse(request.body.read)
      rescue JSON::ParserError
        halt 400, "JSON is required: '#{request.body.read}'"
      end

[bhatfield ~]$ curl -X POST --data "444" http://ubuntu-scratchbox.vbx:5000/publish/junk
JSON is required: ''

HTTP POST Test Failed

Hello,
Sorry to bother you, i'm not able to post any HTTP request to Backstop Proxy.
Maybe you could help me, thanks in advance.

Here are my configs:

Exports :

graphite-01:/opt/backstop# echo $CARBON_URLS
10.10.7.238
graphite-01:/opt/backstop# echo $PREFIXES
test1,test2

Foreman start ok :

graphite-01:/opt/backstop# foreman start
12:58:59 web.1  | started with pid 1762

Test Script :

#!/usr/bin/env ruby -rubygems
require 'rubygems'
require 'sinatra'
require 'rest_client'

# Send a metric
RestClient.post("http://10.10.7.238:5000/publish/test1",
   [{:metric => "test", :value => "123", :measure_time => Time.now.to_i}].to_json)

Client Error view :

graphite-01:/opt/backstop# ruby test.rb 
/usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!': 500 Internal Server Error (RestClient::InternalServerError)
    from /usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:230:in `process_result'
    from /usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in `block in transmit'
    from /usr/local/rvm/rubies/ruby-2.1.0/lib/ruby/2.1.0/net/http.rb:853:in `start'
    from /usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:172:in `transmit'
    from /usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:64:in `execute'
    from /usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient/request.rb:33:in `execute'
    from /usr/local/rvm/gems/ruby-2.1.0/gems/rest-client-1.6.7/lib/restclient.rb:72:in `post'

Server Error view :

12:59:48 web.1  | SocketError - getaddrinfo: Name or service not known:
12:59:48 web.1  | 10.10.7.238 - - [09/Jan/2014 12:59:48] "POST /publish/test1 HTTP/1.1" 500 196146 0.1024

CVE-2013-0263

Heaven detected that rack is not ~> 1.1.6, ~> 1.2.8, ~> 1.3.10, ~> 1.4.5, ~> 1.5.2

Your Gemfile.lock on the master branch currently is 1.4.1.

Can you guys fix this up? 💞

/cc github/security#67 @atmos

Update newrelic_rpm gem to 3.5.3.25

Heaven detected that newrelic_rpm is not >= 3.5.3.25

Your Gemfile.lock on the master branch currently is 3.3.4.1.

Can you guys fix this up? 💞

/cc github/security#76 @atmos

Reconnect to carbon-cache on broken pipe error

If I restart the carbon daemon, the next time data is submitted to backstop it starts throwing 500 errors because of Errno::EPIPE at publisher.rb line 16: broken pipe.

It seems like attempting to reconnect there would be fairly straightforward? A simple approach of 'try to reconnect on every error' should probably work, the overhead of trying is relatively low compared to the system load generated by actually getting a metric in the door.

installing backstop

Hi
I've attempted to install backstop into an Ubuntu 15 vm where I have Graphite also installed. I followed the steps with the following settings:

$ cd /tmp/
git clone https://github.com/obfuscurity/backstop.git
$ cd backstop
$ bundle install
$ export CARBON_URLS= "carbon://localhost:2003"
$ export PREFIXES= test,system_io_by_size,system_totals
$ foreman start

All works as expected so far. How ever when I try to formulate a rest post command the example provided indicates backstop is listening on port 443, but I find no process that is running on this port. I found the process kicked off after foreman start is listening on port 5000 but rest commands to this port return this:

command I run from within my vm:
graphite@jeffs-ubuntu:~/backstop$ curl --data '{"metric" :"test" , "value" : "100", "measure_time" : "2016-07-01"}' http://localhost:5000/publish/note

error returned:

RuntimeError: missing PREFIXES
/home/graphite/backstop/lib/backstop/config.rb:4:in env!' /home/graphite/backstop/lib/backstop/config.rb:10:inprefixes'
/home/graphite/backstop/lib/backstop/web.rb:97:in block in <class:Application>' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:incall'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in block in compile!' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in[]'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in block (3 levels) in route!' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:inroute_eval'
/var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/sinatra.rb:139:in route_eval_with_newrelic' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:inblock (2 levels) in route!'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in block in process_route' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:incatch'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in process_route' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/sinatra.rb:119:inprocess_route_with_newrelic'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in block in route!' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:ineach'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in route!' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:inblock in dispatch!'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in block in invoke' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:incatch'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in invoke' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:indispatch!'
/var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/sinatra.rb:164:in dispatch_and_notice_errors_with_newrelic' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/sinatra.rb:150:inblock in dispatch_with_newrelic'
/var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:363:in perform_action_with_newrelic_trace' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/sinatra.rb:147:indispatch_with_newrelic'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in block in call!' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:inblock in invoke'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in catch' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:ininvoke'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in call!' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:895:incall'
/var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/rack/agent_hooks.rb:30:in traced_call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/rack/browser_monitoring.rb:32:in traced_call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/head.rb:13:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/show_exceptions.rb:25:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:182:in call' /var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:incall'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in call' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:inblock in call'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in synchronize' /var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:incall'
/var/lib/gems/2.1.0/gems/newrelic_rpm-3.16.0.318/lib/new_relic/agent/instrumentation/middleware_tracing.rb:96:in call' /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/tempfile_reaper.rb:15:incall'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/lint.rb:49:in _call' /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/lint.rb:37:incall'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/showexceptions.rb:24:in call' /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:incall'
/var/lib/gems/2.1.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in call' /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/chunked.rb:54:incall'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/content_length.rb:15:in call' /var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/connection.rb:86:inblock in pre_process'
/var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/connection.rb:84:in catch' /var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/connection.rb:84:inpre_process'
/var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/connection.rb:53:in process' /var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/connection.rb:39:inreceive_data'
/var/lib/gems/2.1.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:in run_machine' /var/lib/gems/2.1.0/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:194:inrun'
/var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/backends/base.rb:73:in start' /var/lib/gems/2.1.0/gems/thin-1.7.0/lib/thin/server.rb:162:instart'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/handler/thin.rb:19:in run' /var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/server.rb:286:instart'
/var/lib/gems/2.1.0/gems/rack-1.6.4/lib/rack/server.rb:147:in start' /var/lib/gems/2.1.0/gems/rack-1.6.4/bin/rackup:4:in<top (required)>'
/usr/local/bin/rackup:23:in load' /usr/local/bin/rackup:23:in

'

ENV:
graphite@jeffs-ubuntu:~/backstop$ echo $PREFIXES
test,system_io_by_size,system_totals

help a noob out?

Thanks in advance.

CVE-2013-0269

Heaven detected that json is not ~> 1.5.5, ~> 1.6.8, ~> 1.7.7

Your Gemfile.lock on the master branch currently is 1.6.1.

Can you guys fix this up? 💞

/cc github/security#71 @atmos

Backstop on Debian Wheezy 7.7

Hi Guys,

When I try to install backstop, I get the following errors with event machine and thin:

Installing eventmachine (0.12.10) with native extensions
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.
....
make
compiling rubymain.cpp
rubymain.cpp: In function ‘VALUE t_connect_server(VALUE, VALUE, VALUE)’:
rubymain.cpp:468:42: error: format not a string literal and no format arguments [-Werror=format-security
rubymain.cpp: In function ‘VALUE t_bind_connect_server(VALUE, VALUE, VALUE, VALUE, VALUE)’:
rubymain.cpp:488:42: error: format not a string literal and no format arguments [-Werror=format-security]
cc1plus: some warnings being treated as errors
make: *** [rubymain.o] Error 1

After some research it seems it's because of a Debian desicion to make -Werror=format-securoty a default compile flag. Changing the Gem.lock file to use eventmachine 1.0.3 and thin 1.6.3 lets backstop compile. However, after I export my carbon location and a test prefix, I use netstat to see if anything is listening on port 80 and nothing..... the foreman start command appears to hang as well.
bkvm4:root# export CARBON_URLS="carbon://carbon.example.com:2003"
bkvm4:root# export PREFIXES=test
bkvm4:root# foreman start
16:09:35 web.1 | started with pid 4741
16:09:35 web.1 | Ignoring ruby directive. This is a feature added to Bundler 1.2.0
16:09:35 web.1 | and higher. Please upgrade if you would like to use it.
16:09:35 web.1 |

^ hangs here, not sure if it's supposed to daemonize or not....

bkosick@bkvm4:~/puppet/modules/salesforce/manifests$ sudo netstat -l | grep http retuens nothing.

Any help would be appreciated.
Thanks,
Brian

Trouble around post data

Trying to post data over an http requester (here the firefox addon "httprequester") and get return 'missing field'.

Backstop is listening on http://10.10.7.238:5000/publish/server (where i can reach sinatra page returning 'Sinatra doesn’t know this ditty')

So i'm trying to post this json data:
[{
":metric": "test",
":value": "123",
":measure_time": "Time.now.to_i"
}]

and get return "missing fields"

I have attached the screenshot

PS: is it possible to have more details for usage of PREFIXES export. (how to use it, what it need to contain)