chiwenzhen/crf-word-segmenter

chinese word segment using CRF++

Word Segmenter

1.Install CRF++

• download CRF++
• install CRF++

./configure
make
sudo make install

• install python package(in CRF++-0.xx/python directory)

It enables python to load binary model from disk.

python setup.py build
sudo python setup.py install

2.Preparing Data

• download backoff2005 data
• unzip the data

3.Convert data to CRF++ format

python conv_format.py icwb2-data/training/pku_training.utf8 train.data

4.CRF++ train

crf_learn -f 3 -c 4.0 CRF++-0.58/example/seg/template train.data model

training time: about 20 minutes

5.CRF++ test

python run_test.py model icwb2-data/testing/pku_test.utf8 test.result

6.Backoff2005 experiment

perl icwb2-data/scripts/score icwb2-data/gold/pku_training_words.utf8 icwb2-data/gold/pku_test_gold.utf8 test.result > a.txt

Reference

http://www.mutouxiaogui.cn/blog/?p=224