zseder/hunmisc

miscellaneous tools/scripts for different NLP related tasks

hunmisc's Issues

Accidental push

@zseder I accidentally pushed my changes to master. Could you check them out and see if I have to change anything (HEAD vs HEAD~2)?

levenshtein(): set weights for operations

Enable the user to assign weights to the three basic operations (insertion, deletion and replacement) in LSD. The default must be the current, hard-coded weights (even though it is different from the original case, where the weight of replacement was 2).

Have a custom weighted levenshtein()

Enable the user to customize the LSD further by giving the function three maps:

• insert_map and delete_map would contain character - weight pairs. Their meaning is: if we insert/delete said character, the cost should grow by the specified weight;
• replace_map would store character x character -> weight pairs; I hope its role is obvious.

I would create a new function so that the original levenshtein() could remain as simple as possible. I would even go as far as to create a Levenshtein class that takes as parameters the above (and all parameters the original has) and that has a distance(s1, s2) method.

Actually, thinking about it now, the new functionality does not make the implementation that much more complex (only a few dict.get's), but would probably make it a bit slower. What do you think?

Speed up levenshtein

These algorithms should not really be written in Python, and calling them many times can make the program rather slow, so any method that helps to speed them up is welcome.

1. Max distance. Now for real :)
2. Use only two lists (+ the marginals).

dbpedia parser

the selection of main category should be rewritten so that it doesn't depend on the order of the entity's lines in the dump

7zip file open

Since pylzma doesn't seem to be working, we should create a wrapper around "7zr" the way we did with gzip in xzip.py

max_distance doesn't work, rolling back...

Since ALL cells in the matrix are computed (which might not actually be necessary, I have to check), we can reach max_distance even though the final result is lower than that. So now it has a new meaning: if max_distance < final result, return that.

setup.py install downloads and compiles liblinear on every run