packetloop / packetpig Goto Github PK
View Code? Open in Web Editor NEWPacketpig - Open Source Big Data Security Analytics
Home Page: www.packetloop.com
Packetpig - Open Source Big Data Security Analytics
Home Page: www.packetloop.com
The command run in packetpig/lib/
with all environment variables set correctly
./run_emr -w -f s3://<my_bucket>/packetpig/pig/examples/snort.pig -r s3://<my_bucket>/packetpig/data/web.pcap -o s3://<my_bucket>/output/ -l s3://<my_bucket>/logging/
results in the following output:
/usr/bin/env ruby lib/elastic-mapreduce-ruby/elastic-mapreduce --create --name "spoons's pig jobflow" --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-daemons --args --namenode-opts=-XX:GCTimeRatio=19 --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args -m,mapreduce.task.timeout=0 --args -m,mapred.task.timeout=0 --args -m,tasktracker.http.threads=40 --args -m,mapred.map.tasks.speculative.execution=false --args -s,mapred.map.tasks.speculative.execution=false --bootstrap-action s3://packetpig/bootstrap.sh --instance-group master --instance-count 1 --instance-type m1.large --instance-group core --instance-count 1 --instance-type m1.large --alive --pig-script s3://<my_bucket>/packetpig/pig/examples/snort.pig --step-name snort.pig --args -p,includepath=s3://packetpig/pig/include-emr.pig,-p,snortconfig=/mnt/var/lib/snort-2931/etc/snort.conf,-p,dnspath=/mnt/var/lib/packetpig/lib/scripts/dns_parser.py,-p,tcppath=/mnt/var/lib/packetpig/lib/scripts/tcp.py,-p,pcap=s3://<my_bucket>/packetpig/data/web.pcap,-p,output=s3://<my_bucket>/output/output-1464276028 --ami-version 2.2 --hadoop-version 1.0.3 --pig-version 0.9.2.2
/usr/bin/ruby-mri: No such file or directory -- lib/elastic-mapreduce-ruby/elastic-mapreduce (LoadError)
./run_emr:146:in `create_cluster': undefined method `[]' for nil:NilClass (NoMethodError)
from ./run_emr:20:in `run'
from ./run_emr:444:in `<main>'
ruby -v
provides ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-linux]
and the elastic-mapreduce
gem is installed. Is the script deprecated by now? Or is this a bug on my side?
Lots of web instructional material that references packetpig use the binning.pig example script and an output file is supposed to be created upon completion of the script. The line that does this is commented out in the main branch. So the user does not see the expected result.
This makes the user wonder if they are doing something wrong.
Good morning,
I'm a little fuzzy on the implementation of PacketPig. Does it permit the storage of all captured data locally or is Amazon S3 the only option? I'm looking for something that is just like PacketPig, but not cloud-based.
Thanks,
I am using Ubuntu 14.04 LTS with proxy server authentiction and installed docker and docker.io and when I run command "docker build packetpig ." from the folder where DockerFile is placed then I am getting following in terminal -
mayank@mayank-pc:~/Downloads/packetpig-master$ docker build packetpig .
Usage: docker build [OPTIONS] PATH | URL | -
Build a new image from the source code at PATH
--force-rm=false Always remove intermediate containers, even after unsuccessful builds
--no-cache=false Do not use cache when building the image
-q, --quiet=false Suppress the verbose output generated by the containers
--rm=true Remove intermediate containers after a successful build
-t, --tag="" Repository name (and optionally a tag) to be applied to the resulting image in case of success
Please help me how to install PacketPig.
I just ran through the installation instructions with the docker container. While the example in the instructions for basic_http worked, basic_summary fails as follows:
root@2390b772f892:/src/packetpig# ./pigrun.py -f pig/examples/basic_summary.pig
['pig -v',
'-x local',
'-f pig/examples/basic_summary.pig',
'-param pcap=data/web.pcap',
'-param output=output',
'-param n=1',
'-param snortconfig=lib/snort/etc/snort.conf',
'-param tcppath=lib/scripts/tcp.py',
'-param dnspath=lib/scripts/dns_parser.py']
Executing pig -v -x local -f pig/examples/basic_summary.pig -param pcap=data/web.pcap -param output=output -param n=1 -param snortconfig=lib/snort/etc/snort.conf -param tcppath=lib/scripts/tcp.py -param dnspath=lib/scripts/dns_parser.py
2014-04-14 00:31:03,151 [main] INFO org.apache.pig.Main - Logging error messages to: /src/packetpig/pig_1397435463143.log
2014-04-14 00:31:03,245 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2014-04-14 00:31:03,521 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for packets: $0, $1, $2, $3, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32
2014-04-14 00:31:03,567 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: summary: Store(file:/tmp/temp-1405130183/tmp-2050501636:org.apache.pig.impl.io.InterStorage) - scope-6 Operator Key: scope-6)
2014-04-14 00:31:03,573 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-04-14 00:31:03,632 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-04-14 00:31:03,639 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-04-14 00:31:03,656 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-04-14 00:31:05,902 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-04-14 00:31:05,925 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-04-14 00:31:05,925 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-04-14 00:31:05,934 [Thread-1] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-04-14 00:31:06,052 [Thread-1] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-04-14 00:31:06,054 [Thread-1] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-04-14 00:31:06,251 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for map tasks
2014-04-14 00:31:06,253 [pool-1-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local_0001_m_000000_0
2014-04-14 00:31:06,296 [pool-1-thread-1] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2014-04-14 00:31:06,302 [pool-1-thread-1] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@64498713
2014-04-14 00:31:06,306 [pool-1-thread-1] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 358584
Input split[0]:
Length = 358584
Locations:
2014-04-14 00:31:06,364 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Map task executor complete.
2014-04-14 00:31:06,365 [Thread-12] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:349)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.pig.builtin.COUNT.exec(COUNT.java:71)
at org.apache.pig.builtin.COUNT.exec(COUNT.java:51)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:245)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:322)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:360)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:290)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:237)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.pig.data.DataBag
at org.apache.pig.builtin.COUNT.exec(COUNT.java:57)
... 18 more
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-04-14 00:31:11,427 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2014-04-14 00:31:11,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-04-14 00:31:11,432 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-04-14 00:31:11,432 [main] INFO org.apache.pig.tools.pigstats.PigStats - Detected Local mode. Stats reported below may be incomplete
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u6 0.8.1-cdh3u6 root 2014-04-14 00:31:03 2014-04-14 00:31:11 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_local_0001 packets,summary MAP_ONLY Message: Job failed! Error - NA file:/tmp/temp-1405130183/tmp-2050501636,
Input(s):
Failed to read data from "file:///src/packetpig/data/web.pcap"
Output(s):
Failed to produce result in "file:/tmp/temp-1405130183/tmp-2050501636"
Job DAG:
job_local_0001
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias summary
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
... 7 more
Details also at logfile: /src/packetpig/pig_1397435463143.log
root@2390b772f892:/src/packetpig# cat pig_1397435463143.log
ERROR 1066: Unable to open iterator for alias summary
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
Below is the command, followed by the output, and the error log.
Thanks
RiverCity
[cloudera@localhost PacketPig]$ pig -x local -f binning.pig -param pcap=web.pcap -param output=output
RESULTS************************************************
Failed Jobs:
JobId Alias Feature Message Outputs
job_local2120343510_0001 bw_grouped,bw_summary,packets,tcp,tcp_grouped,tcp_summary,udp,udp_grouped,udp_summary MULTI_QUERY,COMBINER Message: Job failed!
Input(s):
Failed to read data from "file:///home/cloudera/PacketPig/web.pcap"
Output(s):
ERROR 2244: Job failed, hadoop does not return any error message
org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:145)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:475)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
_Vesrion_***************************************
[cloudera@localhost PacketPig]$ hadoop version
Hadoop 2.0.0-cdh4.3.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.3.0/src/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155a
Compiled by jenkins on Mon May 27 19:45:25 PDT 2013
From source with checksum a4218d77f9b12df4e3e49ef96f9d357d
This command was run using /usr/lib/hadoop/hadoop-common-
Hi there,
Following the install doc to the letter I get this. Persists after boot and I'm short on skill to solve it. Sorry to raise an issue rather than send a solution.
Details below:
Command line:
packetpig@packetpig-VirtualBox:~/Documents/packetpig$ pig -x local -f pig/examples/binning.pig -param pcap=data/web.pcap -param output=output
2013-01-14 12:55:17,579 [main] INFO org.apache.pig.Main - Logging error messages to: /home/packetpig/Documents/packetpig/pig_1358168117566.log
2013-01-14 12:55:17,797 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-01-14 12:55:18,391 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-01-14 12:55:18,543 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-01-14 12:55:18,838 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/packetpig/Documents/packetpig/pig_1358168117566.log
From /home/packetpig/Documents/packetpig/pig_1358168117566.log
ERROR 2999: Unexpected internal error. org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple
java.lang.ClassCastException: org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple
at org.apache.pig.pen.DerivedDataVisitor.evaluateOperator(DerivedDataVisitor.java:356)
at org.apache.pig.pen.DerivedDataVisitor.visit(DerivedDataVisitor.java:253)
at org.apache.pig.impl.logicalLayer.LOSplit.visit(LOSplit.java:116)
at org.apache.pig.impl.logicalLayer.LOSplit.visit(LOSplit.java:40)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:70)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:94)
at org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:86)
at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:105)
at org.apache.pig.PigServer.getExamples(PigServer.java:1155)
at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:630)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:308)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
I request that the install instructions for Ubuntu and Mac OS X be expanded to support CentOS. Both Cloudera and Hortonworks deliver VM versions of their Hadoop releases based on a CentOS 6 container and to make life easy for experimentation and learning about PacketPig, it would be nice if a set of steps based on the yum package manager were available.
I have built two different systems a debian 7 and an ubuntu 12.04 and I get the EXACT same error at both places in finishing this, maybe I missed something - twice ?
Bob@packetpig:~/Documents/packetpig$ sudo snort -c lib/snort-2931/etc/snort.conf -A fast -y -l out -r data/web.pcap
Running in IDS mode
--== Initializing Snort ==--
Initializing Output Plugins!
Initializing Preprocessors!
Initializing Plug-ins!
Parsing Rules file "lib/snort-2931/etc/snort.conf"
ERROR: lib/snort-2931/etc/snort.conf(0) Unable to open rules file "lib/snort-2931/etc/snort.conf": No such file or directory.
Fatal Error, Quitting..
Hello all,
Running a 3 node cluster. Local execution of packetpig (http.pig for example) works fine on all 3 nodes. When trying to run a larger pcap(~5GB) through in distributed mode the python scripts (tcp.py and dns.py) seem to only run on one node, causing timeouts on the job. Job timeout is set to 600 secs. Smaller pcap files seem to finish with no issues and produce usable data. Is there a way to distribute the python job across the cluster?
Thanks
Mike
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.