This is a tool to perform Lucene search with inter/intra parallelism implementations.
-
You can clone the repo with the following command:
$ git clone git://github.com/ylwang/lucene-search-parallelism.git
-
Once you've cloned the repository, change directory into
lucene-parallelism-core
and build the package with Maven:$ cd lucene-search-parallelism/lucene-parallelism-core $ mvn clean package appassembler:assemble
-
Build index on the entire collection:
$ sh target/appassembler/bin/IndexStatuses -collection {collectionPath} \ -index {indexPath} -optimize
-
Divide the collection into equal size and build index on each of them:
$ sh target/appassembler/bin/PartitionIndex -collection {collectionPath} \ -index {partitionedIndexPath} -parts {# of partitions} -optimize
-
Run Lucene search with inter parallelism:
$ sh target/appassembler/bin/RunQueriesInterQuery -index {indexPath} \ -queries {queryPath} -threads {# of threads}
-
Run Lucene search with intra parallelism:
$ sh target/appassembler/bin/RunQueriesIntraQuery -index {partitionedIndexPath} \ -queries {queryPath} -threads {# of threads}