coltekin/trmorph

A finite-state morphological analyzer for Turkish

TRmorph (http://www.let.rug.nl/coltekin/trmorph/)

This is the README file for the new version of TRmorph (updated 2015-11)

TRmorph is a open source/free morphological analyzer for Turkish. The current version is a complete rewrite of the (earlier) SFST version of TRmorph using xfst/lexc. This version is in active development, which may cause some problem and incompatibilities between updates. However, the status is a lot better than the older SFST version, and you are recommended to use the new version. The SFST version is still available through the web page above, but it is not developed any further.

TRmorph is being developed with foma. It should also be trivial to compile it using HFST tools as well (since HFST uses foma as back end). Compiling with Xerox tools should also be possible (if you have to) with minor modifications regarding limited reduplication implemented using foma's _eq().

What is new in this version:

• This is a complete rewrite using more-familiar Xerox languages lexc/xfst.
• A completely new lexicon, semi-automatically constructed using web corpora and online dictionaries. (more work is needed, though)
• A revised tag set.
• A few more utilities: stemmer, unknown word guesser, segmenter, and a hyphenation tool.
• A manual, in progress, but it is already usable.
• New license: this version of TRmorph is distributed under MIT License (see the file LICENSE).

If you use this analyzer in your research, and want to cite it, please cite the appropriate papers from the following list:

• Çağrı Çöltekin (2010). A Freely Available Morphological Analyzer for Turkish In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC2010)
• Çağrı Çöltekin (2014) A Set of Open Source Tools for Turkish Natural Language Processing In: Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)

Getting started

You can get the latest version of TRmorph from GitHub here: https://github.com/coltekin/TRmorph. The best is to clone the repository using git, and pull often since this version is changed relatively frequently, but GitHub also allows you to download the as a .zip file.

The compilation requires foma and a C preprocessor (gcc preprocessor is used by default), and make, and a few more UNIX tools. Assuming you have foma[1] installed, type make to compile the analyzer. If all goes well, you should have a binary automaton in foma format called trmorph.fst. After that you can use interactive foma, or flookup for batch processing (both are part of foma). Here are some examples:

$ foma
...
foma[0]: regex @"trmorph.fst";
2.3 MB. 53564 states, 149484 arcs, Cyclic.
foma[1]: up okudum
oku<v><past><1s>
foma[1]: down oku<v><past><2s>
okudun
foma[1]: exit
$ echo "okudum" |flookup trmorph.fst 
okudu   oku<v><past><1s>

There are also separate automata for segmentation, stemming (or lemmatization) and hyphenation that you can compile and use.

See doc/trmorph-manual.pdf for more information.

trmorph's Issues

compiling with HFST tools

Wondering if you have any pointers / instructions for compiling the latest version of TRmorph with HFST tools. These instructions from Apertium are for the SFST version. Similarily, these pre-compilied HFST turkish resources are based on trmorph-0.2.1. It appears an older version is also being used by the HFST web demo -- as it does not have nearly the coverage of your current system.

If possible, I would like to build trmorph.hfst or trmorph.hfst.ol (an hfst transducer in optimized lookup format) from a recent version of TRmorph for testing. It's unclear whether to use hfst-xfst for this (with something like hfst-xfst source ...?), hfst-lexc, hfst-foma, or something else. Any guidance you could provide would be very much appreciated.

Compiling TRMorph on Mac OSX gives error

I compiled foma successfully and having issues with TRMorph on Mac.

I try to compile TRMorh on Mac OSX but getting problem. Can you help me?

orph.lexc:892:22: error: invalid preprocessing directive
#;
^
morph.lexc:947:22: error: invalid preprocessing directive
#;
^
morph.lexc:982:18: error: invalid preprocessing directive
#;
^
morph.lexc:995:18: error: invalid preprocessing directive

make problem

I can't build. trmorph.fst, here's the error message when I "make":

foma -f analyzer.cpp.xfst
Root...make: *** [trmorph.fst] Segmentation fault (core dumped)

Any ideas while making fails ? Thanks in advance.

Possbile Memory leak

We are using TRmorph to stem Turkish words, We have SOLR plugin that communicates with TRmorph server through UDP socket so that SOLR server will obtain root words. But memory used by flookup keeps increasing and never stops. I guess there is a memory leak in flookup code. Have you experienced this issue before?

Below, you see it is using 0.021 tb memory and CPU usage also is very high.

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10841 kangadm 20 0 0.162t 0.021t 792 S 64.8 23.0 7:18.35 flookup

Thanks

Erol Akarsu

TRmorph can't analyse the word "dairemizdeyiz"

It analyses "dairemizdeyim" successfully though.
I'm trying it on the web demo page.

verbs with -mIş are not analyzed as verbal adjective

From: Jonathan North Washington
Subject: [Apertium-turkic] trmorph issue with -mIş
To: apertium-turkic [email protected]

We have a problem with the following phrase

"...doğmamış bebeğimi öldürdü"
"...killed my unborn child"

The form doğmamış is being used as a verbal adjective here, but the
only potential parse being given by trmorph is the following finite
form:
^doğmamış/doğ<t_narr><3s>$

Is there a way to get this correctly parsed as a verbal adjective?

Transitivity marking on verbs

It would be good to be able to mark verbs as transitive (tv) and intransitive (iv) in Trmorph. Here are some files with stems categorised.

URL: https://apertium.svn.sourceforge.net/svnroot/apertium/incubator/apertium-tr-az/dev/trans0200
URL: https://apertium.svn.sourceforge.net/svnroot/apertium/incubator/apertium-tr-az/dev/trans0400
URL: https://apertium.svn.sourceforge.net/svnroot/apertium/incubator/apertium-tr-az/dev/trans1000

(noun)<N><la><V><cont> problem

Foma, version 0.9.18alpha

foma[0]: regex @"trmorph.fst";
7.1 MB. 199995 states, 463870 arcs, Cyclic
foma[1]: down destek<N><la><V><cont>
destekleyor
[...]
foma[1]: up destekliyor
???
foma[1]: up destekleyor
destek<N><la><V><cont>
destek<N><la><V><cont><3s>

<LN> $passive$

the special char , used just in $passive$ (vinfl.fst) overgenerates

-luk is not getting analyzed

As i know -l{U}k suffix is noun making suffix. The words which ends with -luk is not getting analyzed.
ex:
büyüklük --> büyük+l{U}k
olumsuzluk --> olumsuz+l{U}k
yoksulluk --> yoksul+l{U}k
zenginlik --> zengin+l{U}k

Having problem with "yetersizliği"

I am testing TRMorph with server mode like this:

flookup -S -A 127.0.0.1 trmorph.fst

When I send "yetersizliği" word from UDP client, server is getting problem:
sendto() failed: Message too long

client hung

Null outputs for some forms stemmed with lemmatization turned on

After compiling stem.fst with STEMMER_LEMMATIZE set to 1, we get what looks like a null result ("+?") for certain inputs.

It appears that the "+?" output happens regularly with infinitival verbs, i.e. forms that already end with "-mek" or "-mak". (I'm not a Turkish speaker, but I did try it on this list of verbs.)

E.g.,

$ echo "kızartmak" | flookup stem10.fst 
kızartmak
kızar<V>
kızartmak
kızart<V>

$ echo "kızartmak" | flookup stem11.fst
kızartmak
+?

$ echo "ölmek" | flookup stem10.fst 
ölmek
öl<V>

$ echo "ölmek" | flookup stem11.fst 
ölmek
+?

$ echo "çalışmak" | flookup stem10.fst 
çalışmak
çalış<V>

$ echo "çalışmak" | flookup stem11.fst
çalışmak
+?

The only difference between stem10.fst and stem11.fst above is the STEMMER_LEMMATIZE setting.

It seems that in most (maybe all?) of these cases the stem11.fst output would otherwise be the same as the input. Perhaps the “+?” output is expected behavior. Could you clarify?

Thanks.

ila/ile different analyses

Are ila/ile forms of the same word ? e.g. an instrumentaly word ? Should they both get cnjcoo/postp analyses ? If so, we should add 'ila' to the postp lexicon:

$ echo -e "ila\nile" | hfst-proc tr-ky.automorf.hfst
^ila/ila$
^ile/il/il<3s>/ile/ile$

Also, should ile/ila have the same lemma ? -- are they variants of the same word ?

Parsing the analysis results

Hi, we are trying to use the analyzer results in a pipeline however we need the suffixes too, so we are trying to parse and convert the results to a style similar to Zemberek's or Oflazer's. Do you have or happen to know that there exist any converter for this? What do you suggest for this task? Thanks in advance.

Is an Apache 2.0 License a possibility?

Since foma was recently re-released under the more commercially friendly Apache 2.0 license, I was wondering if there's any chance you might consider a release of TRmorph under the same.

comments in lexicon files

It should be possible to have comments in the lexicon files so that we can select entries based on a bilingual wordlist.

Here is a post to the mailing list about it.

http://thread.gmane.org/gmane.science.linguistics.turkic.mt/101

The most likely analysis is invalid for "Defne'm"

Analysis for the word Defne'm:

defne<N><p1s>
Defne<N:prop><p1s>
...

The results are the same for "Defne'm" and "Defnem". The apostrophe should be taken as a cue that we deal with a proper name.