Implementation of bag (ArrayIndexOutOfBoundsException)

[haifengl]

Implementation of bag (ArrayIndexOutOfBoundsException)

[haifengl]

Hi Mike,

Thanks for the bug report! I would like to do c and a. The "training bag" is actually not for training. It is the feature list, and thus should be a set of unique words. But for sanity check, we should check the duplicates in the constructor. I will do these right now. Can you please add a unit test (in the smile.feature.BagTest of test directory) with your test case? Thanks!

Haifeng

[haifengl]

Hi Mike,

The fix is in the master now. Thanks!

[Xyclade]

Thanks! I'll add a unit test and PR it to this issue later today, as I'm in a meeting right now.

[Xyclade]

PS. the reason I found this was by extracting a feature array for 1 category, then for a second one and combining the two feature arrays into 1 new one, causing duplicates to occur as some terms where in both categories. I should probably remove the intersecting features to make the algorithm predictions better, or does the implementation take these into account?

[haifengl]

Yes, you better use Set class to get the union of the features rather than just concat in general. But the constructor now handles the duplicates. So it should be fine for you to use the feature list in your example.

The main problem is the documentation of Bag. It is not for training purpose. It just assumes that you have a set of unique features and uses them to calculate the double valued feature vectors of some text for you.