Giter Site home page Giter Site logo

flume-load-gen's Introduction

Load testing and validation tools for Flume.

At the moment, it caters to TCP-based syslog load testing.

Files:
bin/start_hammer.sh: start script for the load generator
bin/stop_hammer.sh: stop script for the load generator
bin/validate.pl: validates that its input appears as sequential, increasing integers
bin/pig_validate.sh: runs validate.pl via pig in streaming mode
bin/sort_results.pl: helper to sort the output of several validate.pl runs concatenated together (only for human readability)
bin/summarize_results.pl: summarizes the output from concatenated validate.pl runs
bin/hammer_aggregate.sh: aggregates and prints snapshot from hammer client logs. Best to run this periodically in a loop.
src/main/pig/validate.pig: pig script to stream validate.pl
src/main/java/org/apache/flume/tcphammer/Hammer.java: load testing tool

Notes:
To build the load generator, just run mvn clean package; it will leave an executable jar in target/
pig_validate.sh will store a bunch of files on HDFS, then you can just do hadoop fs -cat hdfs://output_dir > results.txt
If you want it to be pretty (sorted) so you can make sense of it, run ./bin/sort_results.pl < results.txt > results-sorted.txt
If you want to get a summary of the results, run ./bin/summarize_results.pl < results.txt | grep -v DEBUG > summary.txt

flume-load-gen's People

Contributors

mpercy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

flume-load-gen's Issues

Maven - Failed to execute goal

Hi, I tried to build flume-load-gen maven package.
But, when I build it using mvn clean package, I got errors like this :

[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:[67,12] error: no suitable method found for info(String,String,int)
[ERROR]
[ERROR] method Logger.info(Marker,String,Throwable) is not applicable
[ERROR] actual argument String cannot be converted to Marker by method invocation conversion
[ERROR] method Logger.info(Marker,String,Object[]) is not applicable
[ERROR] actual argument String cannot be converted to Marker by method invocation conversion
[ERROR] method Logger.info(Marker,String,Object,Object) is not applicable
[ERROR] actual and formal argument lists differ in length
[ERROR] method Logger.info(Marker,String,Object) is not applicable
[ERROR] actual argument String cannot be converted to Marker by method invocation conversion
[ERROR] method Logger.info(Marker,String) is not applicable
[ERROR] actual and formal argument lists differ in length
[ERROR] method Logger.info(String,Throwable) is not applicable
[ERROR] actual and formal argument lists differ in length
[ERROR] method Logger.info(String,Object[]) is not applicable
[ERROR] actual and formal argument lists differ in length
[ERROR] method Logger.info(String,Object,Object) is not applicable
[ERROR] actual argument int cannot be converted to Object by method invocation conversion
[ERROR] method Logger.info(String,Object) is not applicable
[ERROR] actual and formal argument lists differ in length
[ERROR] method Logger.info(String) is not applicable
[ERROR] actual and formal argument lists differ in length
[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:[137,25] error: incompatible types
[ERROR]
[ERROR] could not parse error message: required: Object
[ERROR] found: long
[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:138: error: incompatible types
[ERROR] curLogIntervalNanosSlept / 1000000,
[ERROR] ^
[ERROR]
[ERROR] could not parse error message: required: Object
[ERROR] found: long
[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:139: error: cannot find symbol
[ERROR] String.format("%.2f", (100 * (double) curLogIntervalNanosSlept / (double) logDelta)),
[ERROR] ^
[ERROR]
[ERROR] could not parse error message: symbol: method format(String,double)
[ERROR] location: class String
[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:140: error: incompatible types
[ERROR] curLogIntervalEventsSent,
[ERROR] ^
[ERROR]
[ERROR] could not parse error message: required: Object
[ERROR] found: long
[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:141: error: cannot find symbol
[ERROR] String.format("%.2f", (double) curLogIntervalEventsSent / ((double) logDelta / 1000000000D)),
[ERROR] ^
[ERROR]
[ERROR] could not parse error message: symbol: method format(String,double)
[ERROR] location: class String
[ERROR] /home/vincentius/Documents/flume-load-gen-master/src/main/java/org/apache/flume/tcphammer/Hammer.java:142: error: incompatible types
[ERROR] eventCount,
[ERROR] ^
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

I think maven use wrong slf4j logger in this case. But, I really don't know how to fix this. Anyone can help? thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.