mjiang89/metapad

Meta Pattern Discovery from Massive Text Corpora

metapad's Issues

Can MetaPAD works on Chinese corpus?

Hi,

I am very interested in how MetaPAD works, and I am interested in how it works on Chinese corpus. However, it seems I can't find some files when I changed them into Chinese corpus, so I need your help, can you tell me what should I do for Chinese corpus? Thank you very much.

when I did your test in Github, I get the result of your test. But when I did test run after change the corpus, it gives the following outputs:

rm -rf bin

mkdir -p bin

g++ -std=c++11 -Wall -O3 -msse2 -fopenmp -I.. -pthread -lm -Wno-unused-result -Wno-sign-compare -Wno-unused-variable -Wno-parentheses -Wno-format -o bin/segphrase_train src/main.cpp

Traceback (most recent call last):

File "metapad.py", line 1166, in

Encrypt(file_output_encrypted,file_output_label,file_output_positive,file_output_key,file_input_entitylinking,file_input_goodpattern,file_input_stopwords,10,LEVEL)

File "metapad.py", line 17, in Encrypt

if sentence[n-1][0] == 'PERIOD': sentence = sentence[0:n-1]

IndexError: list index out of range

Traceback (most recent call last):

File "metapad.py", line 1166, in

Encrypt(file_output_encrypted,file_output_label,file_output_positive,file_output_key,file_input_entitylinking,file_input_goodpattern,file_input_stopwords,10,LEVEL)

File "metapad.py", line 17, in Encrypt

if sentence[n-1][0] == 'PERIOD': sentence = sentence[0:n-1]

IndexError: list index out of range

=== Current Settings ===

Iterations = 2

Minimum Support Threshold = 30

Maximum Length Threshold = 20

POS-Tagging Mode Disabled

Discard Ratio = 0.050000

Number of threads = 15

Auto labels from knowledge bases

    Labeling Method = ByLengthByPositive

    Max Positive Samples = 100

    Negative Sampling Ratio = 2

=======

of total tokens = 6438

of total word tokens = 6438

max word token id = 1564

of documents = 810

of POS tags = 0

The number of sentences = 1

unigrams inserted

of frequent patterns of length-1 = 1566

of frequent patterns of length-2 = 2

of frequent patterns of length-3 = 2

of frequent patterns of length-4 = 2

of frequent patterns of length-5 = 2

of frequent patterns of length-6 = 2

of frequent patterns of length-7 = 2

of frequent patterns of length-8 = 2

of frequent patterns of length-9 = 2

of frequent patterns of length-10 = 2

of frequent patterns of length-11 = 2

of frequent patterns of length-12 = 2

of frequent patterns of length-13 = 2

of frequent patterns of length-14 = 2

of frequent patterns of length-15 = 2

of frequent patterns of length-16 = 2

of frequent patterns of length-17 = 2

of frequent patterns of length-18 = 2

of frequent patterns of length-19 = 2

of frequent patterns of length-20 = 2

of frequent patterns = 1584

total occurrence = 35810

feature extraction done!

=== Generate Labels ===

matched positives = 0

matched negatives = 19

selected positives = 0

selected negatives = 19

Loaded Truth = 19

Recognized Truth = 19

Feature Matrix = 1584 X 14

of threads = 15

Start Classifier Training...

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree![ERROR] empty node in decision tree![ERROR] empty node in decision tree!

[ERROR] empty node in decision tree![ERROR] empty node in decision tree![ERROR] empty node in decision tree![ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

cp: cannot stat `cseg/tmp/quality_phrases.txt': No such file or directory

=== Current Settings ===

Iterations = 2

Minimum Support Threshold = 30

Maximum Length Threshold = 20

POS-Tagging Mode Disabled

Discard Ratio = 0.050000

Number of threads = 15

Auto labels from knowledge bases

    Labeling Method = ByLengthByPositive

    Max Positive Samples = 100

    Negative Sampling Ratio = 2

=======

of total tokens = 6438

of total word tokens = 6438

max word token id = 1564

of documents = 810

of POS tags = 0

The number of sentences = 1

unigrams inserted

of frequent patterns of length-1 = 1566

of frequent patterns of length-2 = 2

of frequent patterns of length-3 = 2

of frequent patterns of length-4 = 2

of frequent patterns of length-5 = 2

of frequent patterns of length-6 = 2

of frequent patterns of length-7 = 2

of frequent patterns of length-8 = 2

of frequent patterns of length-9 = 2

of frequent patterns of length-10 = 2

of frequent patterns of length-11 = 2

of frequent patterns of length-12 = 2

of frequent patterns of length-13 = 2

of frequent patterns of length-14 = 2

of frequent patterns of length-15 = 2

of frequent patterns of length-16 = 2

of frequent patterns of length-17 = 2

of frequent patterns of length-18 = 2

of frequent patterns of length-19 = 2

of frequent patterns of length-20 = 2

of frequent patterns = 1584

total occurrence = 35810

feature extraction done!

=== Generate Labels ===

matched positives = 0

matched negatives = 19

selected positives = 0

selected negatives = 19

Loaded Truth = 19

Recognized Truth = 19

Feature Matrix = 1584 X 14

of threads = 15

Start Classifier Training...

[ERROR] empty node in decision tree![ERROR] empty node in decision tree![ERROR] empty node in decision tree![ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree!

[ERROR] empty node in decision tree![ERROR] empty node in decision tree!cp: cannot stat `cseg/tmp/quality_phrases.txt': No such file or directory

Traceback (most recent call last):

File "metapad.py", line 1236, in

SalientFast(file_output_salient,file_output_key,file_input_phrase,file_input_mapping)

File "metapad.py", line 1148, in SalientFast

fr = open(file_input_phrase,'rb')

IOError: [Errno 2] No such file or directory: 'output/top-token-phrase.txt'

Traceback (most recent call last):

File "metapad.py", line 1236, in

SalientFast(file_output_salient,file_output_key,file_input_phrase,file_input_mapping)

File "metapad.py", line 1148, in SalientFast

fr = open(file_input_phrase,'rb')

IOError: [Errno 2] No such file or directory: 'output/bottom-token-phrase.txt'

Thank you