tdiggity/mhagiwara

Automatically exported from code.google.com/p/mhagiwara

mhagiwara's Issues

tinysegmenter.py

        self._patterns = {
            u"[一二三四五六七八九十百千万億兆]":u"M",
            u"[一-龠々〆ヵヶ]":u"H",
            u"[ぁ-ん]":u"I",
            u"[ァ-ヴーｱ-ﾝﾞｰ]":u"K",
            u"[a-zA-Zａ-ｚＡ-Ｚ]":u"A",
            u"[0-9０-９]":u"N"
            }
        self._chartype = []
        for pat in self._patterns:
            regexp = re.compile(pat)
            self._chartype.append([regexp, self._patterns[pat]])

is wrong. The "for in" statement does not preserve written order of the 
dictionary self._patterns.  Since the patterns in self._patterns overlap each 
other, the matching order is significant. In Python 2.7.2, 
u"[一-龠々〆ヵヶ]" precedes 
u"[一二三四五六七八九十百千万億兆]" in the resulted 
self._chartype.

Original issue reported on code.google.com by [email protected] on 10 Jun 2012 at 4:44

tinysegmenter.py

self._UW1 = {u",u":156,

is wrong. It should be:

self._UW1 = {u",":156,

Original issue reported on code.google.com by [email protected] on 10 Jun 2012 at 4:39