hitalex/plsa

PLSA implementation via EM algorithm

This is a PLSA (Probabilistic Latent Semantic Analysis) implementation via the EM (Expectation-Maximization) algorithm.

Current issues:

1. The code are not well tested, so it may contain bugs. The test text are in the folder ./texts and ./test.
2. The code seems not working well with small datasets, such as ./test

Reference:

EM introduction: http://blog.tomtung.com/2011/10/em-algorithm

PLSA introduction: http://blog.tomtung.com/2011/10/plsa

My lda-with-gibbs repo

Note:

A Tutorial on Probabilistic Latent Semantic Analysis by Liangjie Hong is not a very good PLSA introduction material. There are some known bugs.

plsa's Issues

main.py 打开大量文件(比如大于1000)，会报出异常：[Errno 24] Too many open files

hitalex，您好。
查看plsa.py中Document 类，发现实例Document类时，都会有一个file handler，当实例大量Document类时，会造成资源爆满。
建议将file open的操作放在split方法里面，并及时关闭，代码如下

class Document(object):
    def __init__(self, filepath):
        self.filepath = filepath
        #self.file = open(self.filepath)
        self.lines = []
        self.words = []
    def split(self, STOP_WORDS_SET):
        file = open(self.filepath)
        try:
            self.lines = [line for line in file]
        finally:
            file.close()
        for line in self.lines:
            ...

how to inference a new doc?

how to inference a new doc ?
like lsa, trian again with the original docs?
or is there any inference method like lda?