Comparing 'likeliness' of some input against trained model

[jcjohnson]

Comparing 'likeliness' of some input against trained model

[jcjohnson]

This is not currently implemented, but it wouldn't be hard to do. During training the RNN learns a language model, which is able to assign a probability to an arbitrary sequence of tokens; by sampling from this language model we can generate new text, but if you have an existing piece of text that you want to score then you can just compute its probability under the language model.

All you need to do is run a forward pass of the trained model on your new piece of text; the probability of that text is (equivalent to) the training loss. To implement this, you'd need to load the model and set it to evaluate mode as in sample.lua; you'll then need to construct a CrossEntropyCriterion and use it to compute training loss as in train.lua.

[ChrisCummins]

Thanks for the prompt and complete answer! I will definitely have a go at implementing this. If successful, would you be open to a pull request with a score.lua script (or some more appropriate name)?

[aliabbasjp]

@ChrisCummins It is already implemented here:
billzorn@69d91a3

@jcjohnson might want to accept a pull request

[ChrisCummins]

Thanks for the pointer @aliabbasjp! Reading through the diff it seems like this fork has departed quite a bit from upstream. I'm assuming the usage would be something like this?

th train.lua -input_h5 <corpus-to-compare-likeness-against> -init_from <checkpoint-trained-on-dataset> -unk 1

Perhaps @billzorn could weigh-in.

[ianni67]

@aliabbasjp sorry for being dumb, but I cannot find, in the code you pointed out, the function or the options to do what @ChrisCummins asked for. Would you please, give us some more information?
Thank you very much in advance.

[edit]: I use to blame students when they don't do their homeworks before asking for help. So I tried and read more carefully each file "*.lua" in the directory. I found that eval.lua seems to do something quite similar to what Chris was asking for, so the solution should be to modify eval.lua in order to make it evaluate the text from an input file instead of evaluating a split of the learned dataset.
Am I right?