Dump out tf-idf document vectors in pyserini

[castorini]

Dump out tf-idf document vectors in pyserini

[lintool]

hi @zeynepakkalyoncu - let's see if we can make this happen?

[lintool]

So, if we just want td-idf values, it'll be fairly easy, we can just compute the td-idf values in Python.

However, if we want accurate BM25 values, we'll have to do something like this:
https://github.com/castorini/anserini/blob/master/src/test/java/io/anserini/integration/IndexerTest.java#L264

... which will be slow.

[mirzaeiyan]

Could you please describe your solution to compute tf-idf values in python?

[lintool]

hi @alirezamirzaeiyan

We just need to port this over to Java:
https://github.com/castorini/anserini/blob/master/src/main/java/io/anserini/index/IndexUtils.java#L219

It's on @zeynepakkalyoncu 's todo list, but if you want to do it yourself, code contributions are welcome!