vertica / graph-analytics----triangle-counting Goto Github PK
View Code? Open in Web Editor NEWUse Big data tools such as Vertica, Hadoop and PIG to count triangles in a graph. Experimentally compare their performance.
graph-analytics----triangle-counting's Introduction
This is a benchmark on an important graph analytics use case --
triangle counting. Read this blog post first (TBD) for the background
description.
We have provided the solutions in Hadoop, PIG and Vertica. You will be
able to test the Hadoop and PIG solutions on the data sets that we
provide. When the Vertica Community Edition is released, you will be
able to test the Vertica solution as well.
********************************************************************************
** CONTENTS
********************************************************************************
README
This file.
build.xml
makefile for ant.
input/few-edges.txt
Sample graph file, contains 36 reciprocal edges and six triangles. This is
an example of the input format required by the following three solutions.
Edges must be {source:long, dest:long} pairs and each edge must have a
reciprocal edges (that is, there must exist a {dest, source}). The input
format is:
<long> <whitespace> <long> <newline>
<repeat>
scripts/triangle_counter.sql
Vertica script to create an edges table, load data (from 'input/few-edges.txt'
by default) and execute a query that computes the number of triangles in
the corresponding graph.
scripts/triangle_counter.pig
PIG script that computes the number of triangles in a graph. The input is an
edges file (described above) and the output is the number of triangles.
src/com/vertica/mr/graphs/TriangleCounter.java
MapReduce code that runs in Hadoop and counts the number of triangles in
a graph. The input is an edges file (described above) and the output is the
number of triangles.
third-party/pig/pig-0.8.1.tar.gz
PIG libraries included for convenience.
third-party/hadoop/hadoop-0.20.2.tar.gz
Hadoop libraries included for convenience.
********************************************************************************
** REQUIREMENTS
********************************************************************************
* Java 6. Please make sure it is in your path and JAVA_HOME is set correctly.
* Ant 1.6.5 or higher.
********************************************************************************
** EXAMPLES
********************************************************************************
Here are the basics. See the build.xml for other targets and full details.
Hadoop:
The following command, when executed from the base directory, runs the hadoop
solution (in standalone mode) on the input/few-edges.txt file and outputs the
result to the ant output, as well as a directory named "result".
$ ant triangle-counter-standalone
PIG:
The following command, when executed from the base directory, runs the PIG
solution (in local mode) on the input/few-edges.txt file and outputs the result
to the ant output.
$ ant pig-triangle-counter-local
********************************************************************************
** PERFORMANCE TESTING
********************************************************************************
Our performance study is based on an input data set of 1.3GB in
size. That data set can be downloaded here:
http://www.vertica.com/benchmark/TriangleCounting/edges.txt.gz
You are welcome to download and use it for your own testing.
Note that the above ant commands use a single-instance setup, and are
unsuitable for large-scale performance testing. The performance
results reported in our blog post are done on a 4-node cluster, where
each node has 12 cores, 96GB of RAM and 10GBit internconnect. The
hardware power is equivalent to 4 nodes from the HP Vertica Analytics
appliance:
http://h18006.www1.hp.com/storage/server-solutions/vertica-analytics-overview.html
graph-analytics----triangle-counting's People
Stargazers
Watchers
Forkers
ccosentino charles-cai mijuhan duyvk albedium codedk davidkuna rc130 alabamamike smartinsightsfromdata sunsure eektguj shengwenliang colinsongf riodpp iq-scmgraph-analytics----triangle-counting's Issues
build fail error
hi, I was trying to execute the hadoop and I am getting an error BUILD FAILED error with the command:
"$ ant triangle-counter-standalone"
The error says that 'src' does not exist. Could you make it more clear what the base directory and source directory should have? Also where should the build.xml be present (in hadoop directory)?
Thanks!
Where is the build.xml file ?
I was guessing it would be in the hadoop tree but the one there knows nothing about triangles, right?
What does the last reducer do?
I think the first two map/reduce process has completed all you need to do. So the last "AggregateReducer" looks like a redundance because it just repeats what the second reducer does. Am I right? Or is there anything I missed?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.