Can I add documentation for you?

[anttttti]

Can I add documentation for you?

[anttttti]

Of course. Thanks for the offer.

I'll do a code clean up after the Mercari competition closes this week. This includes fixing bugs, and removing differences to sklearn API, for example how .fit() and .transform() behave. It would be a good time to document the code at that point.

[JanzenLiu]

Sure thanks. I would also start working on this after the competition ends. Besides, Are you interested in adding BM25 module? i.e. similar to WordBag but the score of each word for each document is BM25 score, it seems better than simple TF-IDF.

[anttttti]

We can do this, it won't be difficult to add. Average document lengths need to be tracked for the BM25 document length normalization, but that's all. BM25 IDF is applied the same as in TF-IDF, it's only the term+length normalization that works different.

Argument options for TF-IDF/BM25 configurations can get messy, so those need to be sorted out. One option would be refactoring the normalizations into a separate class, that could be used on any set of sparse matrix features.

[JanzenLiu]

That's great. By the way how can I contact you except through Github? I will be better if we can contact more instantly. My email is [email protected], and if you use Facebook or something else it will also be okay for me.