Tokenizer improvements

[commonsearch]

Tokenizer improvements

[Sentimentron]

One thing that's occured to me is that Python 2's re module isn't fully Unicode aware. Choosing some examples from Wikipedia's page on this:

>>> _RE_WHITESPACE.split(u']\u2029[')
[u']\u2029[']

Whereas in Python 3's interpreter:

>>> _RE_WHITESPACE.split(u']\u2029[')
[']', '[']

Back in Python 2, the split method actually works better:

>>> u']\u2029['.split()
[u']', u'[']

[sylvinus]

Right. One more reason not to use simple regexes for this :)

[Sentimentron]

I've just become aware of the NLTK's nltk.tokenize.casual module, which might be appropriate for this job.