choosewhatulike/chinese-ngram-lm-hadoop

A distributed chinese n-gram language model implementation for train and test on large corpus , using Hadoop MapReduce.

Chinese N-Gram Language Model MapReduce

A distributed chinese n-gram language model implementation for train and test on large corpus , using Hadoop MapReduce. The language model treat every chinese character as a "word", thus a tri-gram contains three characters. Trained model can be used in web search and other application with no struggle.

Compile

We use Maven to build our project.

Run

Train

Suppose you have compiled the project, having a jar named lm.jar. Now run the jar with training data path /input/train.dat, output path /output and the gram window size is 3(aka tri-gram). Note your input and output path must be HDFS paths, and input text must be encoded in UTF-8.

$ hadoop jar lm.jar train 3 /output /input/train.dat

You can find the trained model in /output/out/.

Test

You can also evaluate your language model with Perplexity metric. Suppose we want to evaluate our tri-gram model above with data in /input/eval.dat

$ hadoop jar lm.jar eval 3 /output /input/eval.dat

You can find the evaluate result in /output/eval.

Run together

You can also run train and test all together.

$ hadoop jar lm.jar all 3 /output /input/train.dat /input/eval.dat