Unexpected behavior of to_bag_of_terms method

[chartbeat-labs]

Unexpected behavior of to_bag_of_terms method

[covuworie]

Hey @bdewilde, great - thanks very much for the clarification. I can switch to the master branch for the time-being whilst waiting for the new release. Feel free to close this one out! I'll let you know if I spot anything else. Also, I may have some enhancement ideas further down the line. Once again, great library!

[bdewilde]

Sounds good. I'm open to feature requests — and PRs! :)

[bdewilde]

Hi @covuworie , you're describing some recent changes to string normalization that are currently in the master branch on github but have not yet been released — textacy v0.3.2 does not have them. So, if you're using v0.3.2 as you wrote, then you wouldn't see the deprecation messages and changed behavior.

If, instead, you're using the latest version in master, then maybe there's a bug — but I don't see it. That said, I think you've misinterpreted what passing a callable in for the normalize param does: it operates on a per-token or per-span basis, converting the objects into a string representation as desired. So, maybe you want all terms uppercased rather than lowercased, or you want to stem them rather than lemmatize. Also, lemmatization typically involves lowercasing, while lowercasing does not involve lemmatization.

Let me know if you're using v0.3.2 or the latest in master, and if what I wrote above clarifies your expectations.

[covuworie]

Hi @bdewilde, thanks very much for the prompt and clear explanation. I was using the textacy 0.3.2 release. I checked the latest code of doc in master and can see the deprecation messages.

I do not think I misinterpreted the purpose of passing a callable to the normalize param. I read the spaCy documentation for Span which says it's a slice from a Doc object. My docs just happen to be very small and the Span is in fact the whole document! As you said, I expected to be able to use the callable to convert the Span (i.e. the whole Doc) into the string representation I desired. I guess this is what line 489 in doc in master is meant to do? Does textacy 0.3.2 have the same line of code or is it only in master?

[bdewilde]

Hey @covuworie , glad we sorted that out — sorry about the confusion! v0.3.2 doesn't let you pass in a callable to the normalize param (which doesn't exist anyway, I suppose), so unless you want to switch to the master branch on github, you're stuck with lemmatize=True or lowercase=True, for now. I'll put out a new textacy release as soon as I have a bit more to include, hopefully by end of January. I've been sidetracked doing PRs on someone else's open-source project... :)

Is there still an unresolved issue, or can I close this out?