rootroo-ltd/pyhfst

Pure Python implementation of HFST

Pyhfst

Pyhfst is a pure Python implementation of HFST. The library makes it possible to use HFST optimized lookup FSTs without any C dependencies. Both weighted and unweighted FSTs are supported.

The library will run on all operting systems that support Python 3.

Installation

pip install pyhfst

Pyhfst can run way faster if you have Cython installed. After installing Cython, you must reinstall Pyhfst

pip install cython
pip install --upgrade --force-reinstall pyhfst --no-cache-dir

Usage

import pyhfst

input_stream = pyhfst.HfstInputStream("./analyser")
tr = input_stream.read()
print(tr.lookup("voi"))

>> [['voida+V+Act+Ind+Prs+Sg3', 0.0], ['voida+V+Act+Ind+Prs+ConNeg', 0.0], ['voida+V+Act+Ind+Prt+Sg3', 0.0], ['voida+V+Act+Imprt+Prs+ConNeg+Sg2', 0.0], ['voida+V+Act+Imprt+Sg2', 0.0], ['voi+N+Sg+Nom', 0.0], ['voi+Pcle', 0.0], ['voi+Interj', 0.0]]

Citation

Please cite the library as follows:

Alnajjar, K., & Hämäläinen, M. (2023, December). PYHFST: A Pure Python Implementation of HFST. In Lightning Proceedings of NLP4DH and IWCLUL 2023 (pp. 32-35).

@article{pyhfst_2023, 
    title={PyHFST: A Pure Python Implementation of HFST},
    author={Alnajjar, Khalid and H{\"a}m{\"a}l{\"a}inen, Mika},
    booktitle={Lightning Proceedings of NLP4DH and IWCLUL 2023},
    pages={32--35},
    year={2023} 
}

pyhfst's Issues

pyhfst does not return the same results as hfst

As described in the header, pyhfst lookup() does not return the same results as hfst ... I got as far as looking at the definition for analyze(), which returns [] if self.state.input_string[0] == NO_SYMBOL_NUMBER. But I have to go. See the screenshot for evidence of the difference in behavior:

pyhfst fails to load basic hfst file (but succeeds on optimized lookup aka hfstol)

pyhfst crashes when loading an hfst file, as shown in the attached screenshot. It succeeds when loading an optimized lookup version of the hfst file. Hfst of course loads both versions without trouble.

I attach the troubled hfst file as well.
morphophonologyclitics_analyze.hfst.zip

This package fails to load my analyzer (but hfst succeeds)

When I call this:

parser = pyhfst.HfstInputStream("name_of_transducer.hfst").read()

pyhfst throws an error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfb in position 1: invalid start byte

The parallel call with hfst works:

parser = hfst.HfstInputStream("name_of_transducer.hfst").read()

I tried opening the transducer as a byte stream and passing that to pyhfst.HfstInputStream(), but that threw an error claiming that bytes, strings or file names were expected, not BufferedReader objects.

C extensions/Cython vs pure python discrepancy

The built wheel in pure python still has the issue I flagged in my previous problem (#3). Apparently the source in the .tgz file is ok (because that is what pip installs and it works on my machine), but for environments that need pure python (like pyodide), when the wheel is accessed, it still breaks.

There's some discussion here: https://app.gitter.im/#/room/#pyodide_community:gitter.im