How to deal with sparse features

[haifengl]

How to deal with sparse features

[haifengl]

Multiple data formats are available, depending on problems and algorithms. What is your problem and what algorithm do you try to use?

[lfomendes]

Hello haifengl,
I want to try some algorithms (naive, logistic, random forests) to analyze twitter posts, so my features will be "contains word 'love'".
There is the Bag option here -> https://github.com/haifengl/smile/blob/master/Smile/src/main/java/smile/feature/Bag.java But i'm afraid that the model and each tweet will contain a lot of features as 0. The double[][] will be too large to process.

I have found the SparseDataset.java and BinarySparseDataset.java but I don't understand hot to use them with the classifiers.

Thanks

[haifengl]

For document classification, I suggest you to use Maximum Entropy classifier (MaxEnt class, http://haifengl.github.io/smile/doc/index.html). Mathematically, it is equivalent to logistic regression. And our implementation supports sparse data. It takes an integer array for features, of which each element is the index of non zero features. Checkout the unit test case for examples.

[lfomendes]

Humm.. I will definitely try that.
I will compare the MaxEntropy with an implementation using feature vectorization using the hashing trick

Thank you very much

[haifengl]

A common mistake in NLP is that use all words in the documents for the features. It is better to do a feature selection first and use this (much) smaller set of words as features (with the Bag class as the helper). BTW, tree based method (e.g. Random Forest) will be very slow if the number of features is too large.