packetloop/packetpig

Packetpig - Open Source Big Data Security Analytics

packetpig's Issues

ERROR 2999: Unexpected internal error. org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple

Hi there,
Following the install doc to the letter I get this. Persists after boot and I'm short on skill to solve it. Sorry to raise an issue rather than send a solution.

Details below:
Command line:
packetpig@packetpig-VirtualBox:~/Documents/packetpig$ pig -x local -f pig/examples/binning.pig -param pcap=data/web.pcap -param output=output
2013-01-14 12:55:17,579 [main] INFO org.apache.pig.Main - Logging error messages to: /home/packetpig/Documents/packetpig/pig_1358168117566.log
2013-01-14 12:55:17,797 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-01-14 12:55:18,391 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2013-01-14 12:55:18,543 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2013-01-14 12:55:18,838 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple
Details at logfile: /home/packetpig/Documents/packetpig/pig_1358168117566.log

From /home/packetpig/Documents/packetpig/pig_1358168117566.log

Pig Stack Trace

ERROR 2999: Unexpected internal error. org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple

java.lang.ClassCastException: org.apache.pig.data.DefaultDataBag$DefaultDataBagIterator cannot be cast to org.apache.pig.data.Tuple
at org.apache.pig.pen.DerivedDataVisitor.evaluateOperator(DerivedDataVisitor.java:356)
at org.apache.pig.pen.DerivedDataVisitor.visit(DerivedDataVisitor.java:253)
at org.apache.pig.impl.logicalLayer.LOSplit.visit(LOSplit.java:116)
at org.apache.pig.impl.logicalLayer.LOSplit.visit(LOSplit.java:40)
at org.apache.pig.impl.plan.DependencyOrderWalker.walk(DependencyOrderWalker.java:70)
at org.apache.pig.impl.plan.PlanVisitor.visit(PlanVisitor.java:51)
at org.apache.pig.pen.LineageTrimmingVisitor.init(LineageTrimmingVisitor.java:94)
at org.apache.pig.pen.LineageTrimmingVisitor.(LineageTrimmingVisitor.java:86)
at org.apache.pig.pen.ExampleGenerator.getExamples(ExampleGenerator.java:105)
at org.apache.pig.PigServer.getExamples(PigServer.java:1155)
at org.apache.pig.tools.grunt.GruntParser.processIllustrate(GruntParser.java:630)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:308)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)

at org.apache.pig.Main.main(Main.java:107)

Unable to install packetpig from posted installation steps

I am using Ubuntu 14.04 LTS with proxy server authentiction and installed docker and docker.io and when I run command "docker build packetpig ." from the folder where DockerFile is placed then I am getting following in terminal -

mayank@mayank-pc:~/Downloads/packetpig-master$ docker build packetpig .
Usage: docker build [OPTIONS] PATH | URL | -
Build a new image from the source code at PATH
--force-rm=false Always remove intermediate containers, even after unsuccessful builds
--no-cache=false Do not use cache when building the image
-q, --quiet=false Suppress the verbose output generated by the containers
--rm=true Remove intermediate containers after a successful build
-t, --tag="" Repository name (and optionally a tag) to be applied to the resulting image in case of success

Please help me how to install PacketPig.

ERROR: lib/snort-2931/etc/snort.conf(0)

I have built two different systems a debian 7 and an ubuntu 12.04 and I get the EXACT same error at both places in finishing this, maybe I missed something - twice ?

Bob@packetpig:~/Documents/packetpig$ sudo snort -c lib/snort-2931/etc/snort.conf -A fast -y -l out -r data/web.pcap
Running in IDS mode

    --== Initializing Snort ==--

Initializing Output Plugins!
Initializing Preprocessors!
Initializing Plug-ins!
Parsing Rules file "lib/snort-2931/etc/snort.conf"
ERROR: lib/snort-2931/etc/snort.conf(0) Unable to open rules file "lib/snort-2931/etc/snort.conf": No such file or directory.

Fatal Error, Quitting..

Install instructions for CentOS would be great

I request that the install instructions for Ubuntu and Mac OS X be expanded to support CentOS. Both Cloudera and Hortonworks deliver VM versions of their Hadoop releases based on a CentOS 6 container and to make life easy for experimentation and learning about PacketPig, it would be nice if a set of steps based on the yum package manager were available.

ERROR 2244 on latest Cloudera VM for packetpig example binning.pig

Below is the command, followed by the output, and the error log.
Thanks
RiverCity

[cloudera@localhost PacketPig]$ pig -x local -f binning.pig -param pcap=web.pcap -param output=output
RESULTS************************************************
Failed Jobs:
JobId Alias Feature Message Outputs
job_local2120343510_0001 bw_grouped,bw_summary,packets,tcp,tcp_grouped,tcp_summary,udp,udp_grouped,udp_summary MULTI_QUERY,COMBINER Message: Job failed!

Input(s):
Failed to read data from "file:///home/cloudera/PacketPig/web.pcap"
Output(s):

---

Pig Stack Trace

ERROR 2244: Job failed, hadoop does not return any error message

org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:145)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
at org.apache.pig.Main.run(Main.java:475)
at org.apache.pig.Main.main(Main.java:157)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
_Vesrion_***************************************
[cloudera@localhost PacketPig]$ hadoop version
Hadoop 2.0.0-cdh4.3.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.3.0/src/hadoop-common-project/hadoop-common -r 48a9315b342ca16de92fcc5be95ae3650629155a
Compiled by jenkins on Mon May 27 19:45:25 PDT 2013
From source with checksum a4218d77f9b12df4e3e49ef96f9d357d
This command was run using /usr/lib/hadoop/hadoop-common-

lib/run_emr LoadError

The command run in packetpig/lib/ with all environment variables set correctly

./run_emr -w -f s3://<my_bucket>/packetpig/pig/examples/snort.pig -r s3://<my_bucket>/packetpig/data/web.pcap -o s3://<my_bucket>/output/ -l s3://<my_bucket>/logging/

results in the following output:

/usr/bin/env ruby lib/elastic-mapreduce-ruby/elastic-mapreduce --create --name "spoons's pig jobflow" --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-daemons --args --namenode-opts=-XX:GCTimeRatio=19 --bootstrap-action s3://elasticmapreduce/bootstrap-actions/configure-hadoop --args -m,mapreduce.task.timeout=0 --args -m,mapred.task.timeout=0 --args -m,tasktracker.http.threads=40 --args -m,mapred.map.tasks.speculative.execution=false --args -s,mapred.map.tasks.speculative.execution=false --bootstrap-action s3://packetpig/bootstrap.sh --instance-group master --instance-count 1 --instance-type m1.large --instance-group core --instance-count 1 --instance-type m1.large --alive --pig-script s3://<my_bucket>/packetpig/pig/examples/snort.pig --step-name snort.pig --args -p,includepath=s3://packetpig/pig/include-emr.pig,-p,snortconfig=/mnt/var/lib/snort-2931/etc/snort.conf,-p,dnspath=/mnt/var/lib/packetpig/lib/scripts/dns_parser.py,-p,tcppath=/mnt/var/lib/packetpig/lib/scripts/tcp.py,-p,pcap=s3://<my_bucket>/packetpig/data/web.pcap,-p,output=s3://<my_bucket>/output/output-1464276028 --ami-version 2.2 --hadoop-version 1.0.3 --pig-version 0.9.2.2
/usr/bin/ruby-mri: No such file or directory -- lib/elastic-mapreduce-ruby/elastic-mapreduce (LoadError)

./run_emr:146:in `create_cluster': undefined method `[]' for nil:NilClass (NoMethodError)
    from ./run_emr:20:in `run'
    from ./run_emr:444:in `<main>'

ruby -v provides ruby 2.2.5p319 (2016-04-26 revision 54774) [x86_64-linux] and the elastic-mapreduce gem is installed. Is the script deprecated by now? Or is this a bug on my side?

Error running basic_summary with docker container

I just ran through the installation instructions with the docker container. While the example in the instructions for basic_http worked, basic_summary fails as follows:

root@2390b772f892:/src/packetpig# ./pigrun.py -f pig/examples/basic_summary.pig
['pig -v',
'-x local',
'-f pig/examples/basic_summary.pig',
'-param pcap=data/web.pcap',
'-param output=output',
'-param n=1',
'-param snortconfig=lib/snort/etc/snort.conf',
'-param tcppath=lib/scripts/tcp.py',
'-param dnspath=lib/scripts/dns_parser.py']

Executing pig -v -x local -f pig/examples/basic_summary.pig -param pcap=data/web.pcap -param output=output -param n=1 -param snortconfig=lib/snort/etc/snort.conf -param tcppath=lib/scripts/tcp.py -param dnspath=lib/scripts/dns_parser.py
2014-04-14 00:31:03,151 [main] INFO org.apache.pig.Main - Logging error messages to: /src/packetpig/pig_1397435463143.log
2014-04-14 00:31:03,245 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: file:///
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
2014-04-14 00:31:03,472 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - pig.usenewlogicalplan is set to true. New logical plan will be used.
2014-04-14 00:31:03,521 [main] INFO org.apache.pig.newplan.logical.rules.ColumnPruneVisitor - Columns pruned for packets: $0, $1, $2, $3, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32
2014-04-14 00:31:03,567 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - (Name: summary: Store(file:/tmp/temp-1405130183/tmp-2050501636:org.apache.pig.impl.io.InterStorage) - scope-6 Operator Key: scope-6)
2014-04-14 00:31:03,573 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2014-04-14 00:31:03,587 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2014-04-14 00:31:03,632 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-04-14 00:31:03,639 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
2014-04-14 00:31:03,656 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2014-04-14 00:31:05,902 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2014-04-14 00:31:05,925 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2014-04-14 00:31:05,925 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2014-04-14 00:31:05,934 [Thread-1] WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-04-14 00:31:06,052 [Thread-1] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2014-04-14 00:31:06,054 [Thread-1] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2014-04-14 00:31:06,251 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for map tasks
2014-04-14 00:31:06,253 [pool-1-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local_0001_m_000000_0
2014-04-14 00:31:06,296 [pool-1-thread-1] INFO org.apache.hadoop.util.ProcessTree - setsid exited with exit code 0
2014-04-14 00:31:06,302 [pool-1-thread-1] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@64498713
2014-04-14 00:31:06,306 [pool-1-thread-1] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 358584
Input split[0]:
Length = 358584
Locations:

---

2014-04-14 00:31:06,364 [Thread-12] INFO org.apache.hadoop.mapred.LocalJobRunner - Map task executor complete.
2014-04-14 00:31:06,365 [Thread-12] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
java.lang.Exception: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:349)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2106: Error while computing count in COUNT
at org.apache.pig.builtin.COUNT.exec(COUNT.java:71)
at org.apache.pig.builtin.COUNT.exec(COUNT.java:51)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:245)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:322)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:360)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:290)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.runPipeline(PigMapBase.java:237)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:232)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigMapBase.map(PigMapBase.java:53)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to org.apache.pig.data.DataBag
at org.apache.pig.builtin.COUNT.exec(COUNT.java:57)
... 18 more
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local_0001
2014-04-14 00:31:06,426 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2014-04-14 00:31:11,427 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local_0001 has failed! Stop running all dependent jobs
2014-04-14 00:31:11,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2014-04-14 00:31:11,432 [main] ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2014-04-14 00:31:11,432 [main] INFO org.apache.pig.tools.pigstats.PigStats - Detected Local mode. Stats reported below may be incomplete
2014-04-14 00:31:11,433 [main] INFO org.apache.pig.tools.pigstats.PigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.2-cdh3u6 0.8.1-cdh3u6 root 2014-04-14 00:31:03 2014-04-14 00:31:11 UNKNOWN

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_local_0001 packets,summary MAP_ONLY Message: Job failed! Error - NA file:/tmp/temp-1405130183/tmp-2050501636,

Input(s):
Failed to read data from "file:///src/packetpig/data/web.pcap"

Output(s):
Failed to produce result in "file:/tmp/temp-1405130183/tmp-2050501636"

Job DAG:
job_local_0001

2014-04-14 00:31:11,433 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias summary
2014-04-14 00:31:11,436 [main] ERROR org.apache.pig.tools.grunt.Grunt - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)
... 7 more

Details also at logfile: /src/packetpig/pig_1397435463143.log

root@2390b772f892:/src/packetpig# cat pig_1397435463143.log

Pig Stack Trace

ERROR 1066: Unable to open iterator for alias summary

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias summary
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:644)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:171)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:147)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:396)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:755)

... 7 more

binning.pig has last line commented out -- this is a regression?

Lots of web instructional material that references packetpig use the binning.pig example script and an output file is supposed to be created upon completion of the script. The line that does this is commented out in the main branch. So the user does not see the expected result.

This makes the user wonder if they are doing something wrong.

Question about PacketPig architecture

Good morning,

I'm a little fuzzy on the implementation of PacketPig. Does it permit the storage of all captured data locally or is Amazon S3 the only option? I'm looking for something that is just like PacketPig, but not cloud-based.

Thanks,

Python scripts only run on single node.

Hello all,
Running a 3 node cluster. Local execution of packetpig (http.pig for example) works fine on all 3 nodes. When trying to run a larger pcap(~5GB) through in distributed mode the python scripts (tcp.py and dns.py) seem to only run on one node, causing timeouts on the job. Job timeout is set to 600 secs. Smaller pcap files seem to finish with no issues and produce usable data. Is there a way to distribute the python job across the cluster?
Thanks
Mike