hanwentao / x-stream Goto Github PK

                                 X-Stream

  What is it?
  -----------

  X-Stream is a graph processing system for analytics on big graphs using a
  single machine. X-Stream is based on the philosophy that sequential access to
  data works best for all types of storage media: main memory, SSD and magnetic
  disk. It exposes the familiar scatter-gather programming model, but is based
  on the principle of streaming data from storage.

  Installation
  ------------

  Please see the file called INSTALL.

  Getting started
  ---------------

  X-Stream expects the input graphs in the binary edge list format. For details
  see the file called TYPES in the format directory. In addition to the edge
  list file, X-Stream expects an ini file that specifies the graph type, the
  number of vertices and the number of edges. Provided with X-Stream are two
  synthetic graph generators - a random graph generator and an RMAT graph
  generator. Thay also generate the needed ini files. For example, to create a
  scale-free graph with 1M vertices and 16M edges run:

    $ rmat --name test --scale 20 --edges 16777216

  This creates a directed graph. To create an undirected graph instead, use
  --symmetric. Note that some algorithms require directed and some undirected
  graphs to work properly.
  
  Directory format/tools contains a number of scripts to convert graphs from
  different formats into the format that X-Stream expects.

  To run any of the currently implemented algorithms in X-Stream, you can use
  the provided example program called benchmark_driver. To see supported
  options run it with -h. For example, to run 10 iterations of pagerank on the
  test graph generated above, invoke benchmark_driver as:

    $ benchmark_driver -g test -b pagerank --pagerank::niters 10 \
      -a -p 16 --physical_memory 268435456

  This will run X-Stream with 16 worker threads and with 256MB of physical
  memory allotted to it. The -a flag tells X-Stream to automatically tune the
  number of partitions and other system parameters. In addition to the number
  of threads and the amount of physical memory available, you might also want
  to tune the per-processor cache size and cacheline size parameters.

  Please note that that if you do not run the program as superuser, you will
  recieve a number of error messages about being unable to "mlock" memory
  pages. These are not fatal errors, *X-stream will execute
  successfully*. However, as X-Stream does its own buffer management, runtime
  numbers can show some variation if the OS decides to page out X-Stream's
  buffers or due to the cost of the initial minor faults on small runs. For
  benchmarking performance, running as superuser is the right thing to do.

  Licensing
  ---------

  Please see the file called LICENSE.

  Python support
  --------------

  X-Stream has experimental Python support -- algorithms written in Python can
  be executed using the X-Stream C++ engine. To turn Python support on,
  uncomment relevant lines in the Makefile. For examples how to write
  algorithms in Python, check algorithms/bfs/bfs.py and
  algorithms/pagerank/pagerank.py.

  Compression
  -----------

  X-Stream also has experimental zlib support for processing compressed graphs.
  To turn it on, uncomment relevant lines in the Makefile.

  Contact
  -------

  Amitabha Roy <[email protected]>
  Ivo Mihailovic <[email protected]>