Word likelihood positive?

[microsoft]

Word likelihood positive?

[feiga]

@boche The real word likelihood should be word likelihood + normalized likelihood. The computation is factorized

[feiga]

@boche Let me try to make it more clear. Computing the total likelihood will need compute on all the data and model, which is prohibitively expensive. In a data-parallel distributed setting, the data is distributed across several workers and model is distributed stored across several servers.

To compute the likelihood efficiently, we decompose the total likelihood to three parts: doc-likelihood(only concern document, can compute without access of global shared model), word-likelihood(only concern about word-topic-table, can compute with only very small part of model) and normalized item in word-likelihood(only concern about the summary row).

Each worker computes the doc-likelihood on part of documents it holds, and word-likelihood of part of model it needs. Thanks to the decomposition, the computation on each worker can be also finished with multi-threads in a data-parallel fashion.

Then the total likelihood on the whole dataset should be the summation of : 1) all the doc-likelihood on each machines. 2) all the word-likelihood of each words in vocabulary. 3) the normalized likelihood.

[tanglizhe1105]

I run nytimes corpus with 1000 topics in a cluster with 10 machines. I summary all word-likelihood and a normalized-likelihood as the whole dataset word-loglikelihood, but the value is still positive.

[feiga]

Summary word-likelihood and normalized-likelihood is OK. But not summary ALL.